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ABSTRACT 

This report explores the relation between image intensity and object 
shape, it Is shown that iaage intensity is related to surface orientation 
and that a variation in image intensity is related to surface curvature. 
Computational methods are developed which use the measured Intensity 
variation across surfaces of snooth objects to de terrain© surface 
orientation. 

In general, surface orientation is not. deteroined locally by the 
intensity value recorded at each image point. Tools are needed to explore 
the problem of deleraining surface orientation fron image intensity. The 
notion of gradient space, popularized by Huffman and Kackworth, is used to 
represent surface orientation. The notion of a reflectance sap, originated 
by Horn, is used to represent the relation between surface orientation and 
image intensity. The image Hessian is defined and used to represent 
surface curvature. Properties of surface curvature are expressed as 
constraints on possible surface orientations corresponding to a given imaye 
point* Methods are presented which embed assumptions about surface 
curvature in algorithns for deternining surface orientation froo the 
intensities recorded in a single view. 

If additional images of the sane object are obtained by varying the 
direction of incident illunination. then surface orientation is determined 
locally by the intensity values recorded at each ioage point. This fact is 
exploited in a new technique called photometric stereo. 

The visual inspection of surface defects in netal castings is 
considered. Two casting applications are discussed. The first is the 
precision investtaent casting of turbine blades and vanes for aircraft Jet 
engines. In this application, grain size is an inportant process variable- 
The existing industry standard for estimating the average grain size of 
metals is implemented and demonstrated on a sample turbine vane* Grain 
size con be computed from the neasurenents obtained in an ioage, once the 
foreshortening effects of surface curvature are accounted for. The second 
is the green sand mold casting of shuttle eyes for textile looms. Hero, 
physical constraints inherent to the casting process translate into 
constraints on the surface topography of cast objects. In order to exploit 
these constraints, it is necessary to interpret features of intensity as 
features of object shape. 

Both applications demonstrate that successful visual inspection 
requires the ability to interpret observed changes in Intensity in the 
context of surface topography. The theoretical tools developed in this 
report provide a framework for this interpretation. 
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I. INTRODUCTION 
This roport explores the relation between image intensity And object shape. 
Tho research strategy is to exploit, as directly as possible, the intensity 
values recorded in an image. In order to extract the inforoatlon contained 
in image intensity, it is necessary to understand both the geometry of 
image projection and the radlonetry of image formation. It Is shown that 
mage intensity is related to surface orientation and that a variation in 
iaafle intensity is related to surface curvature- 
Basic tools are needed, Gradient space, popularized in [Hurrman 71) 
and [Mackworth 73], is used to represent surface orientation. The 
reflectance nap, introduced in [Horn 77], is used to represent the relation 
between surface orientation and image intensity. With these tools, a fixed 
scene i 1 lumination. object photooetry and imaging geometry can be 
incorporated into an explicit model that allows image intensity to be 
related directly to surface orientation. This relationship is not 
functional since surface orientation has two degrees of freedom and iaago 
intensity provides only one measurement. 

Nevertheless, properties or surface curvature can be expressed as 
constraints on possible surface orientations corresponding to a given image 
point. The image Hessian is defined and used to represent surface 
curvature. Computational methods are presented which enbed assumptions 
about surface curvature in algorithns. for determining surface orientation 
fron the intensities recorded in a single view. 

If additional images of the same object are obtained by varying the 
direction of incident illumination, then surface orientation is determined 
locally by the intensity values recorded at each image point. This fact is 
exploited in a new technique called photometric stereo. 
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Interpreting Image Intensity In terns of underlying object shape Is 
the key to the automatic visual inspection of metal castings. Two casting 
applications are discussed. The first Is the precision investment casting 
of turbine blades and vanes for aircraTt Jet engines. In this application, 
grain size Is an important process variable* The existing industry 
standard for estimating the average grain size of metals is Implemented and 
demonstrated on a saaple turbine vane. Grain size can be computod froca the 
measurements obtained in an image provided that the foreshortening effects 
of surface curvature can be accounted for* Foreshortening can bo accounted 
for once the underlying object shape is known. 

The second Is the green sand moid casting of shuttle eyes for textile 
loons. Here, physical constraints inherent to the casting process 
translate into constraints on the surface topography of cast objects. In 
order to exploit these constraints, however, it is necessary to interpret 
features of intensity as features of object shape. 

Both applications demonstrate that successful visual inspection 
requires the ability to Interpret observed changes in intensity in the 
context of surface topography. The theoretical tools developed in this 
report provide a framework for this interpretation. 

II IMAGE ANALYSIS VERSUS SCENE ANALYSIS 

Vision is a tough problem. One thing that scons to make vision such a 
tough it tih ifii Is the fact that many different kinds of knowledge can 
influence the interpretation of an image. An important question to ask 
concerns how much of the interpretation of an image is forced by the data 
itself and how much can be attributed to the influence of prior 
expectation* This is a difficult question to deal with in human perception 
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because it is impossible to factor out experimentally one effect from the 
other. At best, one can characterize hunan vision as the interaction of 
diverse, partially complete and possibly redundant knowledge sources. 

In machine vision, the distinction between "data forced" and "prior 
expectation" can be dealt with by considering vision to be a two stage 
process. One begins with the intensity values recorded in an iiaage. Image 
analysis extracts features from the raw intensity values and converts these 
features into a convenient symbolic representation. Scene analysis 
interprets the symbolic features produced by image analysis according to 
soma externally defined goal. Image analysis defines what can be 
considered as forced by the data in the subsequent interpretation. Scene 
analysis, on the other hand, is an exercise in problem solving* In scene 
analysis, one is free to invoke whatever prior knowledge is available to 
aid in image interpretation. 

Early artificial intelligence research in machine vision concentrated 
on images of scenes containing plane-faced polyhedra. Initially, the 
distinction between image analysis and scene analysis seemed clear. The 
purpose of iraagc analysis was to generate a two-dioensional line drawing of 
the scene [BinTord and Horn 73], The purpose of scene analysis was to 
interpret a two-dimensional line drawing in terns of the three-dimensional 
objects which gave rise to it [Roberts 651, [Guzaan 68], [Huffnn 71J, 
[Clowes 71], [ttackworth 73], [Waltz 75], [Winston 75]. 

As the field matured, the actual distinction between image analysis 
and scene analysis became less clear, A richer form of interaction between 
image analysis and scene analysis was achieved [Falk 72], [Winston 73], 
[Freuder 76]. One benefit of this new interaction was a reduction in the 
computation roquired in image analysis. Sensitive line-finding procedures 
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existed but generated too many false targets to be applied uniformly over 
an image. Nevertheless, these procedures could be effectively applied to 
verify the presence of lines hypothesized by scene analysis routines 

[Shirai 751. 

■ 

A conceptual distinction between image analysis and scene analysis 
persists. A prevailing notion is that the nature of the features extracted 
from an image is of less consequence than the subsequent interpretation of 
those features. The failings of image analysis are to be compensated for 
by "smarter* scene analysis. Research is lead away from basic image 
analysis towards more elaborate scene analysis- Avoidino imago analysis is 
often not so «uch a question of research philosophy as it is one of 
practical necessity. Yet* it seems critical to determine what information 
can be extracted in imago analysis before attempting to make a distinction 
between image analysis and scene analysis for a particular problem. 

Recent work by [Harr 76] and [Horn 77J has demonstrated that there is 
a great deal of information about three-dimensional shape contained In 
image intensities and that this information can be computed without 
recourse to higher-level knowledge. [Horn 77 J, for example, shows how a 
careful analysis of the intensity profiles in images of polyhedra can be 
used to interpret certain two-dincnsional lines directly as convex, concave 
or occluding edges. Adding this information directly to a line drawing 
produced in image analysis would nake much subsequent scene analysis 
superfluous. 
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1.2 IMAGE ANALYSIS IS HARD 

fluch of the work in Machine vision docs not explicitly exploit the 
information contained in the intensity values recorded in an image. In 
this section, reasons are suggested why image analysis is hard. The 
purpose is to point out some of the difficulties associated with 
interpreting image intensities. Only by taking cognizance or these 
difficulties is it possible to understand when and why various machine 
vision techniques will work and when and why they will fail. 

m THERE ISA LOT OF DATA IN AN IMAGE 

One of the major stunbl ing blocks to image analysis is the sheer 
quantity of data present in an image. One goal of image analysis Is to 
extract features of intensity that are iaportant and to throw everything 
else away- Inage analysis often becomes an exercise in data compression. 
But, the features one can use in a data compression process are those which 
can be conveniently defined in torus of properties of images. There is not 
always a simple correlation between properties of images and properties of 
the objects which give riso to those images* 

As a rough rule of thumb, one can say that practical vision systems 
exist only for domains which have the following two properties: 

■ There is a staple correlation between properties of 
interest in the domain and properties of images of the 
domain. 

■ These image properties ^rc simple to compute. 

A good correlation between properties of the domain and properties of 
images of the domain occurs in domains that are inherently two-dioensional. 
A significant reduction both in the quantity of data present in an image 
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ami in the complexity of feature computation occurs In dooains In which 
intensity can be considered to have only two values. Typical applications 
of machine vision systems involve images, often binary, of two-dimensional 
worlds - Examples are: optical character recognition (OCR); blood cell 
analysis [Young 69]; detection of etching defects in printed circuit 
boards [Ejiri et at 73], 

HUE IMAGE PROJECTION CAUSES PROBLEMS 

Images are defined in two dimensions while objects exist in a 
three-dimensional world. Information is lost In the projection of a 
three-dimensional object onto a two-dimensional image- The sapping from 
object space to ioage space is many-to-one. It is impossible to analyze 
two-dimensional images without specific assumptions about the 
three-dimensional nature of the objects that gave rise to them- A 
particular object feature may appear quite differently depending on the 
viewing direction. More insidious is the fact that projection lntroducos 
two-dimensional image features which have no direct correlation with any 
thrne-dlmnnsional object property. Neighboring points in an image do not 
necessarily correspond to neighboring points on objects. Parts of one 
object may obscure parts of another* 

143 IMAGE ILLUMINATION CAUSES PROBLEMS 

The same object shape viewed with the same imaging geometry will 
generate different image intensities depending on the direction and nature 
of the incident illumination , Changes in illumination nay cause a 
particular object feature to appear Quite differently even when seen from 
the same viewing direction. 
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1.2.4 SURFACE PHOTOMETRY CAUSES PROBLEMS 

The same object shape viewed with the same ln&ging and light source 
geometry will generate different image intensities depending on the surface 
phoioaetry of the object material. Surface photometry varies from objoct 
material to object material. For a fixed object material, surfaco 
photometry varies depending on whether the object is wet or dry, clean or 
dirty. 

1.2.5 THE HUMAN VISUAL SYSTEM IS REMARKABLY FORGIVING 

The human visual system does a remarkable job of interpreting image 
intensity despite problems caused by projection, illumination and surface 
photometry outlined above- At first glance, this might be cited as 
evidence that image analysis cannot intrinsically be hard. This is false 
optimism and soon leads to serious difficulty. 

Designing an image analysis system by letting intuition decide what 
the intensities ought to be guarantees failure. It is necessary to obtain 
digital images and to examine the actual numbers recorded. Flore 
importantly, image analysis requires good physical models of how surfaces 
reflect light, 

IJ INSPECTION IS A GOOD DOMAIN FOR IMAGE ANALYSIS 

Normal visual acuity does not guarantee success as an inspector. The 
Initial problem in inspection is to learn how to interpret the visual data. 
Then, however, the problem becomes one of developing an ablility to 
eliminate the influence of prior expectation on visual Interpretation. 
Inspection can be likened to proofreading text. If one reads sentences, 
the expectation of what a word should be makes it very difficult to catch 
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typographical errors* To proofread effectively, it is necessary to Ignore 
the semantic context of what one is reading and concentrate instead on the 
individual words and letters* Thus, an inspector Is trained to see what he 
is forced to see rather than what he expects to see. This is precisely 
what makes inspection a difficult job for people. This is also what makes 
inspection the right job to hand over to machine vision systems. 

In machine vision, the influence of prior expectation can be carefully 
controlled. In visual inspection,- It is necessary to rely on the 
information contained in the actual intensity values present In an image. 
This makes inspection a good domain for exploring what can and cannot be 
determined from intensity data alone. Inspection forces one to deal 
explicitly with the Issue of what interpretation is forced by the data. 

M METAL CASTING IS A GOOD DOMAIN FOR INSPECTION 

There are important reasons why metal casting Is a good domain for 
research in automatic inspection. One set of reasons relates to the demand 
for automatic inspection systems: 

■ A foundry is a hostile work environment. 05HA and EPA 
regulations are forcing foundries to modernize their 
equipment and safety practices. But making foundries 
more acceptable to humans is costly. A better Idea is to 
automate humans out of hazardous areas* A formidable 
stumbling block to the automatic foundry Is the need for 
inspection. 

■ Strong foreign competition is forcing the casting 
industry to re-evaluate its materials usage and energy 
costs. The goal is to minimize overdcslgn and associated 
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materials waste by achieving tighter control over the 
manufacturing process, Tighter control requires more 
inspection. 

■ Productivity in casting inspection is low* Flaintalnlng 
tight quality standards is a difficult and costly 
operation. Inspection is a tedious job for humans. 
Tasks requiring close visual attention create fatigue. 
Repetitive tasks induce boredom. Yet inspection demands 
mental and physical alertness. Automation is the key to 
increasing inspection productivity, 

■ Casting is a vital component of the metal working 
industry- The trend is towards the increased Importance 
of casting as a primary fabrication process. More 
sophisticated casting reduces the number or expensive 
machining operations required to complete the part, Horo 
dependable casting reduces the number of expensive 
machining operations wasted on defective parts. 
Sophisticated and dependable casting requires better 
quality inspection. 

Another set of reasons for why metal casting is a good domain for 
research in automatic inspection relates to the need for flexible, 
computer-based systems: 

■ Foundries are job-shop environments. The typical foundry 
casts many different part geooetries in small to medium 
production quantities. The specialized techniques 
available for the automation of largo volume production 
are not suited to the foundry. The palletized, pick and 
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place, orientation preservin-g style of parts handling 
envhiinmii by most autoaation engineers is the antithesis 
of the parts handling problen in the foundry. Castings 
are tumbled freely in coercion shaker machines to break 
away residual mold material and gating. Castings are 
cleaned in common sand-blast machines* It Is not 
feasible to redesign these operations to preserve part 
identity and orientation. On-line inspection must be 
flexible enough to accommodate different part geometries 
and different methods of part presentation- 

m The inspection of a casting does not result in a simple 
PASS/FAIL decision- Inspection requires interpretation. 
Often, this interpretation depends on different kinds of 
knowledge. Knowledge of the casting process determines 
the nature of the defects to be found. Knowledge of 
subsequent machining operations determines the 

acceptability of surface iaperfections. Knowledge of 
in-service stress patterns determines the critical 
tolerances for each subsection of the part. In order to 
apply existing standards, autonatic inspection systens 
must be able to interpret test data within the context of 
the part as a whole. 

■ The inspection of castings remains something of an art. 
Often, there is only a loose correlation between test 
results and hard evidence as to what functional 
characteristics of the part are actually being measured. 
Current research in nondestructive testing enphasi2es the 
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extraction of more information rrom existing tost data as 
much as it does the development of new test techniques- 
The ability to store and retrieve test data on a computer 
system adds a new dioenslon to nondestructive testing. 
Computer-based interpretation is the key to exploiting as 
much of the information in the test data as possible. 

■ Foundries depend critically on the experience of their 
inspectors. One consequence of this dependence is that 
there is little or no industry-wide standardization of 
test interpretation. There is a strong desire to develop 
such standards for casting Inspection- Autooation of 
test interpretation Is one way to achieve 
standardization. Flexibility in autooation is the key to 
making such standardization useful. 

■ There is no casting free of imperfections. Imperfections 
are inherent to the casting process. The goal of 
manufacturing is to hold these imperfections to specified 
tolerances. In the autooation of casting inspection, the 
emphasis must be away from simple detection devices and 
towards computer-based systens. The goal is to allow 
maximum flexibility in the specification and 
interpretation of Inspection standards* 
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1.5 THE RESEARCH APPROACH 

Several questions require attention: 

■ How are inages formed? 

■ How does the real world constrain an imago? 

■ How Is it possible to exploit those constraints In 
order to interpret the intensity values recorded In an 
image? 

One can begin by writing down an equation describing the image-forming 
process as follows: 

*tmago intensity* = <incident illumination* x <surface reflectance> (1.5.1) 
The terms image intensity, incident illumination and surface reflectance 
must be made precise, but for the moment, the technical details will be 
omitted in order to make soae qualitative observations. In writing down 
this equation, enphasis is being placed on the fact that imago Intensity 
can be treated as a measurement arising from a real physical process . 
Inages can be interpreted by understanding the underlying physical process 
which gave rise to them. 

Four factors interact in image formation: 

■ imaging geonotry 

■ incident illumination 

■ surface photometry 

■ surface topography 

The imaging geometry determines the projection of the 
three-dimensional object space coordinates onto two-dimensional image 
coordinates. Without illumination, there can be no image. Incident 
illumination is characterized by the spatial and spectral distribution and 
state of polarization of radiant energy falling on the object surface. 
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Surface photometry determines how light reflects off the object surface. 
It is determined by the optical constants of the object oaterlal and by the 
surface microstructure. (Surface micros tructure is surface detail which Is 
too fine to be rosolved in the image but which causes observable effects in 
the way light is reflected at the object surface.) Surface topography is 
surface detail which is within the resolution limits of the imaging 
hardware. It characterizes the gross object shape relative to the viewer. 
Equation (1.5.1) relates these four factors. The notions of gradient 
space and the reflectance nap allow this equation to be made precise. For 
fixed surface photometry, incident illumination and iaaging geonetry, this 
equation is used to study how to determine surface topography froa image 
intensity. 

The work presented in this report is based on a physical model of how 
images are formed. Wo attempt is made to account for the subjective 
phenomena associated with human perception. [Becfc 74]. for example, 
contains an image of a ■matte" vase, B-y adding two local ■specularitios'' 
to this image, a new picture is created in which the entire vase appears 
shiny- Regions in the first image appear dull while the same regions in 
the second image appear shiny, even though the intensity values recorded 
are identical. Equation (1.5.1) accounts only for the intensity values 
recorded in an inago and not for the huivan perception of those values. 

I.5J RELATED APPLICATIONS 

There are a variety of uses for equation (J. 5.1), In computer 
graphics, one probleo is the generation of gray-level images from surface 
models. Using an assumed surface photometry, incident illumination and 
imaging geometry. (1.5.1) can be used to synthesize images of objects. 
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Since gray-level images are readily interpreted by humans, this is a useful 
tool in such diverse fields as computer aided design and automated 
cartography. Work in computer graphics has lad to a number of 
phnnomenological models of surface reflectance {Phong 75]. 

If the surface topography, incident illumination and imaging geometry 
arc known, then (1.5.1) can be used to deternine surface photoootry from 
image intensity. If the spectral nature of incident illumination or the 
spectral response of the imaging hardware is varied* then the dependence of 
surface photometry on the wavelength of incident llluaination can be 
measured* This has been applied to determining ground cover fro« 
miilti spectral 1-ANDSAT iragery registered to digital terrain models 
[Horn & Bachman 77]. 

If the surface topography, incident illumination, isaglng geometry and 

surface photometry are known, then (1.5.1) can be used either to determine 
the optical constants of the object material, assuming a known surface 
micros t rue ture, or to determine surface micros t rue ture* assuming known 
optical constants. Reflectance spectroscopy uses (1.5.1) to deteraine 
optical constants. This field has become an important new tool in analytic 
chemistry- There are oany natcrUls that cannot be analyzed by traditional 
spectroscopic methods. Some of these materials, however, can be ground 
into fine powders of known particle size and shape. In turn, analytic 
models have been developed to relate the surface photometry of such powders 
to the optical constants of the object material of which they arc composed 
[Wendlandt & Hecht 66]. 

Another example analyzes how the observed pattern of sunlight (or 
moonlight) glitter on a wind-ruffled sea can be used to deduce information 
about the sea state (Pl«s et at 77]. This technique is proposed both as a 
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method for evaluating models of ocean wave structure and for Baking 
measurements of the parameters that deternine sea roughness, such as wind 
speed and direction- Analyzing the reflection of light froa a flat water 
surface is a simple exercise in the geometry of specular reflection. Waves 
introduce surface microstructure. The observed glitter pattern consists of 
numerous instantaneous •glints" produced by perfect specular reflection 
from wave facets having the appropriate momentary orientation. The shape 
of the glitter pattern is determined by the distribution of wave 
Inclinations on the water surface. Determining this distribution requires 
that the light source and iDaging geometry be -known since these introduce 
modifications in the apparent wave-slope distribution* 

The above examples illustrate the importance of explicitly accounting 
for the dependence of image intensity on the Tour factors: surface 
photometry, surface topography, incident illunlnatlon and imaging geometry. 
In moit applications, it is convenient to standardize one or more of these 
factors in order to determine the dependence of image intensity on the 
others. 

1.6 SUMMARY OF THE REPORT 

Chapter I introduces the research- A distinction between image 
analysis and scene analysis is made. Reasons are suggested for why image 
analysis is hard, Nevertheless, image analysis can benefit from an 
explicit model of the imago-fortiing process. 5uch a nodel must account for 
the effects or surface photometry. surface topography, incident 
illumination and magi rig geoaetry. Automatic inspection is presented as a 
good domain Tor image analysis- fletal casting is presented as a good 
domain for automatic Inspection. 
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Chapter Z Introduces a framework for Interpreting the intensity values 
recorded in an image* Image analysis is formulated as the problem of 
determining, at each imago point, the range, the surface orientation and 
thr surface curvature at the corresponding object point* Viewer-centered 
representations for surface orientation and surface curvature are 
developed. Gradient space is used to represent surface orientation. The 
image Hessian matrix is used to represent surface curvature. The 
reflectance map establishes the relationship between surface orientation 
and mage intensity. The image Hessian matrix establishes the relationship 
between movement in the image and change in surface orientation. Image 
intensity constrains the image Hessian matrix but does not determine It 
completely. Assumptions about surfac-c curvature further constrain the 
Hessian matrix. These results are demonstrated in a particular relaxation 
algorithm for determining surface orientation from a single view. 
Propagation of local constraint over regions of the image is used to assign 
a global interpretation to the surface. The algorithm is illustrated using 
a synthesized image of a sphere and the assumption that it is a convex 
solid of revolution. 

Chapter 3 develops the relationship between surface curvature and 
image intensity in more detail. Known properties of surface curvature can 
simplify image analysis. A number of cases are considered. These are 
surfaces with constant image Hessian, surfaces that are singly curved and 
surfaces of generalized cones. These special cases illustrate when shape 
information can bo determined solely from image Intensity and when shape 
information can be determined only when additional information is provided, 
either implicitly, in the form of assumptions about surface curvature, or 
explicity from other sources, such as object boundaries. Finally, if 
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additional images of the same object arc obtained by varying the direction 
of incident il lunmalion. then surface orientation can be determined 
locally by the intensity values recorded at each lnage point. This fact is 
exploited in a new technique called photometric stereo. 

Chapter 4 provides a brief introduction to the probleo of »etal 
casting inspection. Two particular casting applications are presented. 
One is the precision investment casting or turbine blades and vanes for 
aircraft Jet engines. The other is the green sattd mold casting of shuttle 
eyes for textile looas. This chapter is not critical to the technical 
development. Rather, it serves as a general introduction to casting 
inspection for the non metallurgist and as a background for the particular 
probleo of grain size estination discussed in chapter 5. 

Chapter 5 presents a program for estimating the average grain size of 
metals based on the three circle (Abrams) procedure defined In ASTM 
Designation; E 112-74 "5tandard Methods for Estimating the Average Grain 
Size of Metals". The program makes use of multiple edge aask filters to 
determine grain boundaries crossed by each circular test pattern. Grain 
crossings are determined by comparing the relative response or three 
different mask sizes. This makes the program usable over a broader range 
of image magnifications and grain sizes and. at the sane time, makes It 
less -susceptible to false indications due to noise. The program Is 
demonstrated using a sample turbine airfoil - An apparent change in grain 
stzo from one region of the image to another may be due to a change in 
grain size- or to a change in the view angle- In order to determine actual 
grain size, it is necessary to explicitly account for the foreshortening 
effect of the view angle at each image point. This can be done if the 
surface orientation is known at each image point- 
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Chapter 6 illustrates the experimental determination of a reflectance 
map for the gray iron used in the green sand nold casting of shuttle eyes 
for textile looms . An improvised photo-goniometer is used to make the 
required reflectance measurements. An inproviscd coordinate gauging 
machine is used to determine object relief. The results are combined to 
produce a synthesized image of the shuttle eye. 

Chapter 7 presents soae concluding remarks concerning the research. 

Appendix A sunoarizes mathematical details that were ommitted in the 
body of the report* 

Appendix B presents a glossary of typical casting defects- 
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% INTERPRETING IMAGE INTENSITY 
This chapter develops a framework for interpreting the intensity values 
recorded in an image. The goal is to relate image intensity to topographic 
properties of the object being imaged. Three such properties are 
considered: 

■ Range 

■ Surface Orientation 

■ Surface Curvature 

Those properties are the basis for a viewer-centered representation of 
object shape. The relation between these properties and linage intensity is 
determined by the surface photonctry or the objects In view, by the naturo 
of the incident illumination and by the object surface, light source and 
viewer geonctry. This relation is aade explicit by exanining the basic 
inage-forming equation: 

<iHge intensity* = -incident illumination* x <surface reflectence> (2-1) 

Analysis or this equation is used to show how iaage intensity relates to 
surface orientation and how changes in lnage intensity relate to surface 
curvature* 

image analysis is foraulated as the probleo of determining, at each 
iaage point, the range, the surface orientation and the surface curvature 
at the corresponding object point. Range is the distance of the object 
point fron the viewer. It is a one-parameter function at oach ioage point. 
Surface or f eatotioJi is the direction of a viewer-facing surface nornal at 
the object point. It is a two-paraneter function at each iaage point. 
Surface curvature is the two principal curvatures, and associated 
directions, at the object point* It is a three-parameter function at each 
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image point. Taken together, range* surface orientation and surface 
curvature define the object rcticf at each image point* 

Analysis of equation (2.1) is facilitated using the notions of 
gradient space and a reflectance nap. Gradient space is a convenient wny 
of representing surface orientation- A reflectance map is a convenient way 
of representing image intensity as a function of surface orientation* With 
those tools, tho basic imagc-forning equation takes on a siople form* 
Constraints ioposed by the light source, object surface and viewer geometry 
are made explicit- The reflectance nap is the basis for phototactric 
methods to determine tho range, surface orientation and surface curvature 
at object points corresponding to selected image points. 

A particular relaxation algorithn Is presented for determining surface 
orientation from a single view by propagation of nutual constraint on 
possible orientations at selected ioage points consistent with general 
hypotheses about object shape* Constraints on surface curvature are 
translated into geometric constraints on possible surface orientations that 
can correspond to a given inage point. This algorithm servos to illustrate 
that the local Interpretation of image intensity depends upon local 
assumptions about surface curvature. 

M A VIEWER-CENTERED REPRESENTATION FOR SHAPE 

In inage analysis, representations used depend more upon what It is 
possible to compute from the intensity values In an lnagc than upon what is 
ultimately desirable- In subsequent stages of analysis t representations 
can be made nore sensitive to the needs of the specific application. 
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A viewer-centered representation is appropriate for the initial 
determination of the shape of objects in an image. The representation 
choson distinguishes the following three classes or image feature: 
1* Points of range discontinuity (occlusion boundaries). 
Z. Points of surface-orientation discontinuity 

(convex/concave edges). 
3, Sections of snoot h surface. 
[Harr 77b] calls this representation the 2-1/2-D sketch and argues for 
its usefulness as an intermediate stage in a vision system for doteraining 
an object-centered description of shape. It is useful to explore how this 
viewer-centered 2-1/2-D sketch can be determined directly from the 
intensity values recorded in an image. Ihe particular question addressed 
In this chapter is how the range, surface orientation and surface curvature 
can be determined over sections of smooth surface. 

2.2 DETERMINING OBJECT RELIEF 

Beforo developing nethods based upon image analysis, it Is worthwhile 
to review other possible techniques for determining object relief at a 
distance. Figure 2-1 summarises the possibilities. 

Roughly speaking, methods for determining object relief can bo divided 
into two categories. First, there are methods which attempt to measure 
range directly. Such methods can be based on tine-of-f light measurements 
of a signal reflected back from the object surface. One design uses a 
pulsed laser to measure time-of-f light. This approach has proven practical 
for the long distance ranging required in space exploration [Johnston 73]. 
It has also boon applied to more terrestrial endeavors [Caulfield 76], A 
more common design for short distance ranging measures tlme-of-f light as a 
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Figure 2*1 Techniques Tor determining object relief. 
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phase shift between the emitted and reluming signal. The CV LIDAR 
Pointing System developed at the C. S. Draper Laboratory measures 
time-of-f light this way using a noncoherent LED centered at 910 nanometers 
and amplitude modulated at SO 11Hz, [DRAPER 73]. This system is reported 
accurate to ±1 millimeter at a total distance of about 3 meters, 
approximately ±1 part in 3000. A similar system has been developed at 
Stanford Research Institute and is reported in [Nitzan et at 77], This 
last reference includes a useful discussion of the inherent design 
trade-offs associated with such direct ranging devices. Basically, 
accuracy is achieved by maintaining a favorable signal to noise ratio. The 
signal to noise ratio can be improved either by increasing the power of the 
source or increasing the time constant used to sanple each point. 

Focussing can also be thought of as a direct ranging technique. Focus 
in an optical system depends on the range of the objects being imaged. 
Techniques for the automatic focussing of imaging systems have been 
explored (Horn 68], In an early experimental systeti [Winston 73], 
automatic focussing was able to locate objects in a scene to an accuracy of 
about ±1 inch at a total distance of about 6 feet, approximately ±1 part in 
100. The use of focussing as a ranging technique depends on the presence of 
sharp features, such as edges, in the scene* It is not suited for ranging 
ovor objects with uniform surface cover and smoothly varying topography. 

Second, there are nethods which determine range fro» indirect 
measurements. These techniques can be further subdivided into 

triangulation methods and photometric nethods. One kind of triangulation 
method uses a stereo pair of images obtained fro* two known positions. Any 
point in an image determines a ray in space. The range at a point in an 
imago is detercined by computing the intersection of this ray and the ray 
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detenained by the corresponding point in the other Image. But, in order to 
determine the second ray, it is first necessary to find the corresponding 
point in the other linage. Thus, stereo ranging suffers from the same 
qualitative limitations as does the technique of automatic focussing. 
Solving the correspondence problem depends on the presence of features In 
one image that can be matched to features in the other. It is unsuitable 
for ranging over objects with uniform surface cover and smoothly varying 
topography. 

Another kind of tnangulation method avoids the correspondence problem 
by using specially controlled illumination. The idea is to project thin 
sheets of light sequentially on the scene and record the image resulting 
from each sheet. The range at a point in an image is deteroined by 
computing the intersection of the ray associated with the image point and 
the sheet of light which llluntnated the point. 

It is also possible to use specially controlled illumination to 
determine surface orientation. (Will & Pennington 71] describe a method 
which uses special illumination to code surface orientation as the 
modulation on a spatial frequency carrier grid. With this method, it Is 
possible to extract the surface orientation of planar regions in a scene by 
linear frequency domain filtering. 

Finally, as indicated above, photometric methods arc methods which 
indirectly determine range from analysis of the image- forming equation 
(H.l). A photometric approach to range determination was first proposed in 
[van Diggelen 51] and subsequently applied to lunar Images obtained froa 
Ranger spacecraft [Rindfleisch 66]. The particular reflectance properties 
of the material of the maria of the moon cause a slnplification In the 
mathematics required to determine range from intensity* [Horn 75] 
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generalized the photometric approach to account for surfaces with arbitrary 
isotropic reflectance properties. Photometric methods determine surface 
orientation from image intensity. Once the surface orientation at each 
object point is determined, range values are obtained by starting at a 
known point and integrating surface orientation over sections of smooth 
surface. Photometric methods are complementary to stereo ranging and focus 
ranging because photometric methods are most suited to objects with uniform 
surface cover and smoothly varying topography. 

Regardless or how object relief is determined, It is important to 
realize that object relief, in the form of a e-l/2-D sketch, corresponds to 
an image with effects due to incident illunination and surface photometry 
explicitly factored out- Thus, exploring how to determine object relief 
directly froia the intensity values recorded in an mage is important to 
image analysis because it addresses the issue of how to account for the 
effects of incident illunination and surface photometry. 

Wote. however, that two stumbling blocks remain. First, determining 
object relieT does nothing to overcome the problem of the quantity of data 
present in an image. Indeed, if object relief is determined at each image 
point, the quantity of data and associated computational load increases 
significantly. But. subsequent scene analysis has a more explicit 
representation to deal with and thus can do a better Job 
[Nevatia * Binford 73] [Narr 77b]- Second, while knowledge of object 
relief solves part of the problen associated with image projection 
(eg. depth discontinuities can be used to separate objects in an image), it 
nevertheless corresponds to a viewer-centered representation of shape. 
Additional assumptions are required in order to recover the 
three-dimensional structure lost in image projection. 
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2-S GRADIENT SPACE AND THE REFLECTANCE MAP 

In order to understand the correspondence between image intensity and 
object roller, it is necessary to relate the geometry of image projection 
to the rarilomotry of image formation* But, understanding the radiooetry of 
image formation requires a model or the way surfaces reflect light. 

The fraction of light reflected by a surface in a given direction 

depends upon the optical properties of the object Material, the surface 

microstructure and the spatial and spectral distribution and state of 

polarization of the incident light. A fcey photometric observation is: 

ffo natter how conptcx the distribution oj incident 
it f iirot Jiatf off, Jor most surfaces, the fraction of the 
incident tight reflected in a particular direction depends 
only on the surface orientation. 

This section introduces the tools required to take advantage of this 

observation. 

The reflectance characteristics of an object material can be 
represented as a function *(J.e,g) of the three angles i t e and 17 defined 
in figure 2-2. These angles arc called, respectively, the incident, 
emergent and phase angles. In this wort, the emergent angle e will also be 
referred to as the view angle. The angles i, e and g are defined relative 
to the object surface. *(i.e.g) determines the ratio of radiance to 
irradiance neasured per unit surface area, per unit solid angle, in the 
direction of the viewer. The reflectance function defined here is related 
to the bi-directional reflectances-distribution function defined by the 
National Bureau of Standards [Nicodemus ct at 77]. 

Inage-forming systems perform a perspective projection as illustrated 
in figure 2-3{a). If the size of the objects in view is small compared to 
the viewing distance, then the perspective projection can be approximated 
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Figure 2-2 Defining tho three angles i. e and g. The incident angle i is the 
angle between the incident ray and the surface normal. The view angle e is 
the angle between the emergent ray and the surface normal. The phase angle 
g is the angle between the incident and eaergent rays. 
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as an orthographic projection as illustrated in figure 2-3(b). 

Consider an loage- forming system that performs an orthographic 
projection. To standardize the imaging geometry, it is convenient to align 
the viewing direction with the negative z-axis. Also, assune appropriate 
scaling of the image plane so that object point (x.y.z) maps onto image 
point (u,v) where: 

u = x 

v c y 

With this imaging geometry, the use of separate image coordinates (u f v) Is 
redundant- Henceforth, image coordinates (x,y) and object coordinates 
(x.y) will be referred to interchangeably. 

If the equation of a snooth surface is given explicitly as: 

* ■ flx.y) 
thnn a surface normal is given by the vector; 



3x fly 



If parameters p and a are defined by: 



p . *rt*.r) 

dx 



dy 

then the surface normal becomes [p,q, -1 J. The quantity (p.q ) will be 
colled the prodicat, and gradient jpgee is the two-dimensional space of all 
such points (p»q)* With the viewer looking along the z-axis, i il q,-l J 
defines a viewer-facing surface normal. 

Gradient space is a convenient way to represent surface orientation. 
It has been used in scene analysis [flackworth 73]. In image analysis. It 
is usod to relate the geooetry of lunge projection to the radiooetry of 
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Perspective Projection 

u * LfJ* v = lj_U 
lz*dl [z*M 




Orthographic Projection 
usx v * y 




Figure 2-3 Characterizing lnage projection. To avoid inage inversion, it Is 
convenient to think of the image plane as in front of the lens rather than 
behind it. Figure 2-3(a) illustrates a perspective projection, where d is 
the distance from the object coordinate origin to tho center of the lens 
and f is the focal length. For objects that are small relative to the 
viewing distance, the inage projection becooes orthographic. In an 
orthographic projection, illustrated in figure Z-3(b), the focal length f 
is infinite so that all rays from object to image plane are parallel. 
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image formation [Horn 77]. This relation is established by showing how 
image intensity is determined by the surface reflectance function 4(l*e,g) 
and how this function can be represented as a function or the gradient 

(P,<t). 

Suppose that each surface element receives a known constant incident 
illumination. Then, *(i,e.g) determines the actual amount of light 
reflected, per unit surface area, per unit solid angle, in tho diroction of 
the viewer. In an orthographic projection* each picture element subtends 
the same solid angle. Tims, *(i»c,g} determines image intensity. 

In an orthographic projection, tho direction to the viewer is constant 
at oach object point. The incident angle l and the view angle e depend 
only on surface orientation* The phase angle g is constant. Thus, 
*(i-o»0>. a function of three variables, can be represented as a function 
of the two gradient coordinates p and <|. For a given distribution of 
incident illumination, a given surface-viewer geometry and a given object 
material, the image intensity corresponding to a surface element with 
gradient (p,q) is unique. 

The re/fect<rffce map R(p,q) determines image intensity as a function of 
p and i). The basic image-forming equation (2.1) becones: 

Hx,y) -= R(p«q) (2.3.1) 

Note what has happened. The reflectance map captures the surface 
pho tone try of an object material for a particular light source, object 
surface and viewer geometry. The two assumptions necessary to write (2-1) 
as (2-3-1) are: 

1. The incident illumination is constant at each surface 
clement. 

2. The inage projection is orthographic. 
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Reflectance mops can bo determined empirically. derived from 
phcnomenological models of surface reflectance or derived from analytic 
models of surTace raicrostructure. The reflectance map is independent of 
the shape or the objects being viewed. It represents explicit knowledge or 
intensities that can be recorded from objects Dade of a given material and 
viewed under a particular light source and viewer geometry. A reflectance 
pap is not an ioage. 

Equation {2,3,1) is the basic equation used to relate image intensity 
to the geometry of the imiigc-rorming process* It is one equation in the 
two unknowns p and q. Thus, the prohlea of determining surface orientation 
from intensity becomes the problcfl of finding the point in gradient space 
(p.q) corresponding to the image point (x,y). 

The simplest case for incident illumination is that of a single 
distant point source. Suppose such a source exists and that object space 
vector [p*,q*.-lj points in its direction. That is t the source is located 
in gradient space at point (p» ,q, ) . With the imaging geometry of 
figure Z-3(b), object space vector [0,0,-1] points in the direction or the 
viewer. That is, the viewer is located in gradient space at point (0,0)* 
Recall that object space vector £p.q,-lj defines a viewer-facing normal at 
surTace point (x f y,z). That Is, (p,q) is the gradient point corresponding 
to surface point (x,y f 2). 

Then, writing the cosine of the angle between two vectors as the dot 
product divided by the product of magnitudes, expressions for cos(l) t 
cos(e) and cos(g) become: 
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cos(i) = ' + PPe * W* 

7l + p, ? 4 q> ? 7i ♦ p ? ♦ q* 



cos(c) = 



7l + p ? ♦ q z 
cos(g} = — 



/I + P* z + Q* z 
Using the above expressions, the transformation of an arbitrary 

surface reflectance function *li,e,g) Into a roflectanco map R(p,q) Is 
straightforward. One simple. Idealized model of surface reflectance Is 
given by: 

*(i,e.g) * & cos(i) 
This reflectance function corresponds to the phanomenological BOdel of a 
perfectly diffuse (lanbertian) surface which appears equally bright from 
all viewing directions. Mere, o is a reflectance factor and the cosine of 
the incident angle accounts for the foreshortening of the surface as seen 
from the source. The corresponding reflectance map is given by: 



Rtp.q) * — Pf + pft ♦ qqQ 



^1 + h* ♦ q« 2 J\ * ; 



It is convenient to present R(p,q) as a series of "iso-brightness - contours 



in gradient space* Figure 2-4 shows the reflectance nap R(p,q) • p cos(i) 
drawn as a series of contours, for the case p« = 0.7, q« = 0.3 and o = 1. 
Specifying a single distant point source Is not a fundamental 
restriction. Nonpoint sources can be node led as the superposition of 
single point sources. The reflectance map does, however, assume equal 
illumination at all surface elements. The reflectance map does not account 
for the fact that certain surface elements can be shadowed with respect to 
one or more of the sources nor for the fact that, for nonconvex objects. 
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Figure 2-4 The reflectance 



map Tor a lambertian surface illuminated by 
single distant point source at gradient p« = 0.7 and q t = 0.3 with n = J 
The reflectance aap is plotted as 
apart. 



series of contours spaced 0.1 units 
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certain surface elenents can receive secondary illumination due to light 
reflected fron other sections of surface (i.e., mutual illumination)- 

2,4 RE-EXAMINING PHYSICAL CONSTRAINTS 

One can formulate the problem of determining the point in gradient 
spaco (p,q) corresponding to the image intensity point I(x.y) analytically. 
The result is a nonlinear first-order partial differential equation which 
can be solved numerically using the method of characteristic strip 
expansion [Horn 75] [Horn 77], For the moment, the formal nature of the 
problem will bo put aside in order to re-examine its basis in the physical 
world. 

By now, it should be clear how points in gradient space correspond to 
orientations in object space. However, it Is possible to be more explicit 
about the correspondence between movement in gradient space and changes In 
surface orientation- 
Using elementary trigonometry, the expression for cos(e) con be 
rewritten as; 

tan(e) ■ vV + Q* 
The inclination of the surface with respect to the viewing direction varies 
mono ton ically with distance from the origin in gradient space. 
Specifically, the distance from the origin is the tangent of the angle 
between the surface normal and the viewing direction. As the gradient 
(p,q) moves away from the origin, the inclination of the surfaco with 
respect to the viewer increases. 

The view angle e characterizes one of the two degrees of freedom 
associated with an arbitrary orientation in space. The locos of points in 
object space having a constant view angle e defines a right circular cone 
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with axis along the viewing direction. The angular position tan -1 <q/p) of 
each gradient (p«q) on the circle p' * q 2 = t*n ? (e) determines the 
direction of steepest descent in inage space along this cone. In general, 
the angular position of a point (p,q) in gradient space corresponds to the 
direction of steepest descent in imago space along the object surface - 
Rotating object space about the viewing direction induces an equal rotation 
in gradient space. 

Two physical constraints can now be identified: 

1, A given point on a physical surface has a unique 
orientation in space, 

2. Matter is cohesive- It is separated into objects. Tho 
surfaces of objects are generally soooth compared with 
their distance from the viewer* 

These are essentially the sane two constraints [ttarr & Poggio 76] use 
as a basis for their- method to compute stereo disparity. As in their 
paper, the problem is to translate the above two physical constraints into 
rules for how points in an image can be natched to points in gradient 
space. 

In their Dost general form, those rules can be expressed as: 

1. UNIQUENESS; 

Each image point may be assigned to at most one location 
in gradient space. 

2. C0WTINU1TV: 

Surfaces vary smoothly almost everywhere- Only a small 
fraction of the area of an image is composed of 
boundaries that correspond to discontinuities of range 
or surface orientation. 

The task ahead is to demonstrate that these rules can be explicitly 

embedded in a computation. The result is an algorithm which attempts to 

achieve a global correspondence between iaage points and points in gradient 

space via propagation of local constraints. 
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Such methods have been called cooperative algorithms 

[ftarr ft Poggio 76J or relaxation labelling [Rosonrcld. Hummel & Zucker 76]. 
Although the implementation has some Intrinsic interest, what is important 
hero U to understand the physical basis for the local constraints used and 
to net the flavor of how these local constraints can propagate back and 
forth to constrain globally possible Hatches between image points and 
points in gradient space* 

2.5 SPECIFYING LOCAL CONSTRAINT 
The basic imago-forming equation 

Kx,y) - R(p.q) 
is one equation in the two unknowns p and q. With this equation alone, the 
gradient corresponding to a particular inage point is constrained to He on 

a ono-parancter family of "iso-brightness" contours in gradient space. The 
goal Is to apply further constraint in order to assign a unique location in 
gradient space to each inage point* 

The essential physical constraint to be exploited is that surfaces 
vary smoothly almost everywhere. This surface smoothness assumption is 
translated into monotonicity rules on changes to view angle and changes to 
direction of steopest descent permitted between closely spaced image 

points. 

One can illustrate how physical constraint adds additional constraint 
to the possible gradient space solutions to the basic equation 
I(x,y) = R(p»<|). Suppose, two closely spaced inage points (xj ,yi ) and 
(xz.y?) are hypothesized to correspond to object points on the same section 
of smooth surface. Further, suppose that the view angle increases In going 
fro* (xi t yi) to (X2,yz) and that the angular position, corresponding to the 



R. J. Woodham 42 Interpreting Image Intensity 

direction or steepest descent, decreases in going from (xi.yi) to (X2,y?). 
Let (pi .Qi ) end (p?.qz) be the gradient locations corresponding to (xj t y\ ) 
and (x?.y 2 ). 

Suppose, further, that Kxj.yi) >ai and l(x*,y?) = a*. Let Ci and C? 
be, respectively, the contours in gradient space given by R(p,q) ■ ar and 
R(P.q) = 02 as shown in figure Z-5, Then. (pi,qi) lies on Ci and (pj # q2) 
1 los on C? 1 

Figure 2-6 shows both the gradient space circle corresponding to the 
maximum view angle interpretation of (P2,q2> (i.e.. tht circle passing 
through the point on C* furthest fron the origin) and the gradient space 
line corresponding to the minimus direction of steepest descent 
interpretation of (p? t q?) (I.e.. the line passing through the point on Ca 
with minimun angular position). Since the view angle is assumed to 
increase in going from (xi.yi) to (X7.y?). the contour of permissible 
(pi »qi ) can be restricted to include only those gradient points on C\ lying 
on or within the circle of the naxlmun view angle interpretation of 
(p?i<!2)- Similarly, since angular position is assuncd to decrease in going 
from (xl ,yj ) to (x;.y2>* the contour or permissible (pj .q> ) can be 
restricted to include only those gradient points on or above the line of 
the minimum direction of steepest descent interpretation or dv.g/). Tluu, 
without any additional constraint on <p2 f q?). the assumed mono tonicity 
relations between (xi,yi) and {x* t y z ) have been applied to the reflectance 
map contours to constrain the possible interpretation of (pi.qi) to include 
only those points of Ci indicated by the solid line of figure 2-6. 
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Figure ?-S The two reflectance nap contours Ci and C?« corresponding to 
R(p.q) = ai and R(p,q) = *2* which det*r«lne possible surface orientations 
at image points <xi ,yi ) and (*?,>'?)- 
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Figure ?-6 The restricted subsection of contour Ci that Is consistent with 
the hypothesis that the view angle increases and the direction or steepest 
descent decreases in going from (xj t yt) to (w,yj). 
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Let ir t 6) denote the polar representation of the gradient (p,q). That 

is: 

r a J\t 2 + q ? 
« tan' ' (q/p) 

Generalizing from the above illustration, the following two rules are 

stated: 

RULE I: CHANGES IK VIEW AKGLE 

Let Ii.I?,...»In be a set of closely spaced image points 
hypothesized to correspond to object points on the same 
section of saooth surface that are nonotonlcally 

nondecreasing in view angle. Let Ci ,C; C* be the 

corresponding set of gradient contours determined from the 
reflectance map. Then, each contour C. can be further 

constrained such that, for each (r,0) I C, , 1 = 2,3 n-1 

mln (r I (r,0) iCi.i) < r < mx |r I (r,tf) iCui) 
Similarly, if [|*If «** + *!« is hypothesized to correspond to 
a set of object points on the sane section of smooth 
surface that are monotonically nonincreasing in view angle, 
then each contour d can bo further constrained such that, 

for each ir,S) iC 1( i » 2,3 n-1 

min (r | (r t tf) i Ci . j } s r s max (r I (r.f) • C. - i > 
RULE II: CHANGES 1M D1HECTI0H OF STEEPEST DESCENT 

Let Ij.l? In be a set of closely spaced image points 

hypothesized to correspond to object points on the sane 
section of smooth surface that are nonotonlcally 
nondecreasing in direction or steepest descant. As above, 
let Ci,C2,....Cn be the corresponding set of gradient 
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contours. Then, each contour C» can be further constrained 
such that, for each (r.*) fC* t i * Z,3,...,n-1 

min Iff | (r,«) iCi| s f < ux {f | (r.#) « C. . i } 

Similarly, if 1 1 . I2 In is- hypothesized to correspond to 

a set of object points on the same section of smooth 
surface that are nonotonically nonincreasing in direction 
of steepest descent , then each contour Ci can be further 
constrained such that, for each (r,«) iC, , 1 » 2,3 n-1 

mln (0 I (r,ff> 1 Q, , ) s 8 4 aax [8 | (r.») «C|.|> 

Z& HYPOTHESIZING MONOTONICITY RELATIONS 

It is now time to turn to the question of how to hypothesize 
roonotonicity relations between selected image points. To begin with, 
consider the worst possible approach* For some snail value of n, one night 
explore all possible ordering*, with respect to both view angle and 

direction of steepest descent, of selected image points Ii.I; U, The 

hope would be that only a small fraction of those orderings would have 
admissable interpretations (i.e.. interpretations that included at least 
one gradient point for each image point). The constraints imposed by each 
interpretation would propagate to neighboring sets of selected image points 
to provide further sutual constraint. Again, the hope would be that 
propagation of local constraint would converge to a correct global 
interpretation while "incorrect" propagations would quickly die out. 

Consider a second possible approach. Suppose a particular surface 
interpretation is forced onto the <Jata. Such an interpretation would 
provide a framework to partially order selected, closely spaced, imago 
points with respect to changes in both view angle and direction of steepest 



R- J, Woodh; 



47 



Interpreting Image Intensity 



descent. Instead of allowing all possible ordering* to compete, this 
second approach pursues a particular interpretation. Again, the hope would 
be that propagation of local constraint would converge to a single global 
interpretation that represents a sinple distortion of the particular 
interpretation being forced. Here, simple distortion implies any surface 
that preserves the assumed oono tonicity relations concerning changes to 
view angle and changes to direction of steepest descent. 

The method actually iraplencnted corresponds to this second approach. 
The program has a swill set of interpretations it is willing to pursue- 
Some are quite rigid, others are quite flexible. In the next section, a 
specific example is presented- For now. a brief analysis is given to show 
how convexity can be used to hypothesize monotonicity relations between 
closely spaced image points. In the process, an important theory will be 
developed. Appendix A contains a wire detailed formulation of the 
mathenatical results presented here. 

By taking partial derivatives of the basic equation I(x,y) = R(p.q) 
with respect to x and y two equations are obtained which can be written as 
the single matrix equation: 



I. 
I* 


= 


Pk q. 







(Throughout, subscripts arB used to denote 
Sioilarly, the two first-order equations: 

dp ■ pidx ♦ p ¥ dy 
dq v q*dx * q*dy 
can be written as the single natrix equation; 



(2.6,1) 



partial differentiation) . 
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dp 


= 






dx 

dy 



(2.6.2) 

For smooth surfaces, the order or differentiation can be interchanged. 
Recalling the original definitions of p and q, this Beans that 

One can define a matrix il by: 



H - 



2 f(x.v> 


dxdy 
3 ? f(x,y) 


ax 2 


axay 


dy 2 



H is the standard Hessian matrix of the function z = f(x,y). More, H Is 
called the image Hessian matrix of the surface 2 « f(x t y), II is a 
viewer-centered representation of surface curvature. In Appendix A. 5, the 
image Hessian matrix H is related directly to an object-centerod 
definition of curvature. 

Equation (2.6.2) can be written in the form; 

[dp.dqr ■ » MXpdtff 
The image Hessian natrix H relates oovenent in the image to the 
corresponding movement in gradient space* 

The particular result to be used in this section is that the 
definiteness of H is related to the convexity/concavity of the object 
surface z a f(x,y): 

The object surface z ' J(x*y) is convex with respect to the 
viewer ij and only if the corresponding image Hessian 
matrix H is positive setnide/tnite. 
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Similarly, 

The object surface z * f(x.y) is concave wtth respect to 
the viewer \J and only xf the corresponding image Hessian 
matrix H is negative semidefinitc. 

Suppose 7 3 f(x.y) is convex. Then, H is positive semidefinitc. 
Multiplying the two matrix equations (2.6.1) and (2.6.2) on the left by 
[R P P<, ) and [dx dy] respectively, gives rise to the two inequalities: 

R* I. + Rq ly 2 (2.6.3) 

dp dx + dcr dy fc (2.6.4) 

Two similar inequalities hold, with the sense of the inequality 

reversed, if z = f(x,y) is concave. Note, however, that concavity need not 

be treated as a separate case. Indeed* ignoring shadows and nutual 

illumination, the classic indentation /protrusion anbiguity has a simple 

expression in this framework. 

// Hx.y) is the tnage corresponding to a concave surface 
z * fix.y} illuminated bu a single point source at gradient 
point Ofetfi) then I(x t y> is vl$o the imge corresponding 
to tAe convex surface z * -f(x t y) illuminated by a single 
point source at gradient point (-p% t -i/*). 

The first inequality (2.6.3) is an additional constraint on the 
gradient corresponding to a point on a convex surface z * f(x,y). The 
normal vector [R P .R«] to the contour of constant reflectance at any point 
(p»q> hypothesized to be a solution to the basic equation l(x,y) = R(p»q) 
must have a nonnogativc conponcnt in the direction of the normal vector 
[Ii . Iy ] to the contour of constant intensity at (x,y)- 

Tfte second inequality (2.6.4} can be viewed a^ an additional 
constraint on the possible movement [dp.dq] in gradient space corresponding 
to a pavement [dx»dy] in the image. The vector [dp.dq] must have a 
nonnegative component in the direction [dx,dy]. In Appendix A. 2, it is 
shown how to choose [dx.dyj to guarantee either the sign of the change to 
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the view angle or the sign of the ehange to the direction of steepest 
descent. 



2.7 ACHIEVINC GLOBAL CONSTRAINT 

Regardless of what mechanism is used to hypothesize oonotonicity 
relations between points in image spues, it is still necessary to embed 
that mechanism in a confutation to achieve global constraint. The 
implementation approach taken here is somewhat ad hoc. The program selects 
nine points in a 3 x 3 square pattern as its basic set of closely spaced 
image points. This pattern serves as the set I, . \ 2 , . , . , ]„ r or applying the 
local constraint criteria (Rules I and II above). First, however, the set 

11 ,l2 ,B ls Passed to the chosen hypothesizing routines to be partially 

ordered with respect to view angle and direction of steepest descent. The 
reflectance nap R(p,q) ls then used to deteroine the initial contour of 
possible gradient space locations for each point 1,. Rules I and II are 
iteratlvely applied to these contours until no further mutual constraint is 
provided. 

The above describes the basic application of local constraint to each 
3x3 square pattern. The selection of successive 3x3 pattorns ls 
allowed to overlap. Thus, each image point I, will eventually belong to 
nine 3x3 patterns. Each time a particular inage point I, is further 
constrained by the application or local constraint to a pattern of which It 
is a monber, each of its eight other patterns is aarked for 
reconsideration. Before novlng on to a previously unconsidered pattern 
local constraint is applied iteratively to each marked pattern, with 
additional marking added as required, until no narked patterns remain to be 
reconsidered. Each time an inage point I, is considered, any additional 
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constraint on the gradient Apace contour of possible solutions to the basic 

equation I(x a y) = R(p,q) propagates through this local filtering mechanism 

to all other image points under consideration. 

The next issue to arise is the question or how to terminate the growth 

of patterns- Currently, the progran terminates on one of two conditions: 
1. One or more of the image points under consideration has 
no admissable gradient space interpretation. This moans 
the algorithm can no longer assign surface orientations 
to image points consistent with the hypotheses about 
surface curvature used to generate the partial orderlngs 
with respect to changes to view angle and changes to 
direction of steepest descent* In this case, the 
Torced" interpretation is dcened to have failed* 
Z. Any new patterns will cross either a contour of range 
discontinuity (OCCLUSION boundary) or a contour of 
discontinuity or surface normal {CONVEX or CONCAVE 
edge). OCCLUSION boundaries arc detected by noting when 
boundary points to existing patterns have a view angle 
greater than some preassigned value. (That is, c Is 
approaching n/2). CONVEX or CONCAVE edges, on the other 
hand, cannot be detected while patterns are being 
expanded. Unless external knowledge is provided about 
the existence of such edges, patterns will propagate 
across discontinuities in surface normal. Depending on 
the nature of the discontinuity, it nay lead to failure 
due to condition 1 above or It oay never be noticed- 
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2.5 AN ILLUSTRATIVE EXAMPLE 

Consider the simple example of a lanbertian sphere illuminated by a 
single distant light source. ThB intensity space to gradient space 
correspondence Hill be derived analytically and used to generate a 
synthetic ioage. The algorithm will be applied to the synthesized imago. 
The results will be used as a basis for Judging the performance of the 
algorithm. 

The first task is to determine the reflectance map. As above, the 
reflectance map for a lambertian surface illuminated rrom gradient point 
(Pb .q. ) is given by: 

R(p.q> = P(l * pPi « q q»> __ 

Vl * Pt 2 * <h' -/l * p 2 * q ? 

For the example, p, = 0.7. q, = 0.3 and p = 1. This reflectance map has 
been illustrated in figure 2-4. 

The socond task Is to determine the surface orientation at oach image 
point. For the exanple. this result is easy to derive. Let the sphere bo 
centered at the object space origin and have radius r. Then, the equation 
of the sphere is given implicitly by; 

x 7 ♦ y 2 ■* z* = r 2 

Elementary calculus Hill verify that the vector tx.y.z] defines an 
outward surface normal at each object point (x.y.z). The corresponding 
gradient is obtained by rewriting this normal as [*x/2.-y/2,-ll. For each 
(x.y). there are two possible 2 values. With the viewer looking along the 
positive z-axis. however, the hemisphere in view corresponds to negative 
values of t. For this imaging geonetry. the explicit equation of the sphere 
is given by: 

z = -vV - x* - y* 
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The synthetic image intensity I(x,y) is equal to the value of the 

reflectance map at the corresponding (p.q). The equation for lnage 

intensity is: 

if x* + y 2 > r 1 

Ux,y> = 

max(O.R(~x/z a -y/z)) otherwise 



where z = -vY z - x 2 - y 2 

Finally, the mechanism by which aonotonicity relations were 
hypothesized for this example nust be specified. One additional assumption 
was used- The algorithm assunos that it Knows at least one image point 
corresponding to an object point oriented directly facing the viewer. Now. 
let such a point define a picudo-oriffin in image space. Since the view 
angle is assumed to be zero at the pseudo-origin, the only possible 
interpretation is that, in a particular direction, the view angle is 
locally nondecreasing with increasing image distance from the 
pseudo-origin. In general, one cannot hope to assert any local 
nonotonicity relation on direction of steepest descent based on angular 
position about the pseudo-origin, tf, however, the surface Is known to bo 
convex, the direction of steepest descent is locally nondecreasing with 
increasing angular position about the pseudo-origin. For the example, the 

set of image points Ii.I; U was ordered in view angle according to 

their distance in imago space fron the pseudo-origin and ordered in 
direction of steepest descent according to their angular position about the 
pseudo-origin. Applied together* these hypotheses *ro equivalent to tho 
strong assumption that the surface in question is a convex solid of 
revolution, with the axis of revolution along the viewing direction. 
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For the example, r » 60. A 128 x 128 synthesized image was generated 
corresponding to the values -64 S x s 63 and -64 S y s 63. The algorithm 
was applied to this image using 3x3 square patterns sampled at an image 
spacing of 5 points in both X and Y. The pseudo-origin was defined as x ■ 
and y = 0. The results are presented as a series of figures. Figure Z-7 Is 
the synthesized inage. Figure 2-6 shows the points in gradient space, 
determined analytically, corresponding to the sampled image points. 
Figure 2-9 redraws figure 2-8 but with the gradient points corresponding to 
the self-shadowed portion of the sphere excluded. All gradient points to 
the left of the contour R(p p q) = correspond to surface points oriented 
Bore than 50° away from the direction of incident illunlnation. 

Figure 2-10 shows the restricted subsection of contour determined by 
the algorithm for each sampled point of figure 2-7. Finally, figure Z-ll 
suporinposes the correct gradient point on each subsection of contour of 
figure 2-10 to illustrate how well the algorithm has performed. The 
crosses mark the correct gradient points, determined analytically, while 
the corresponding subsection of contour marks how well the algorithm has 
isolated those points. 




Figure 2*7 The synthesized wage of a sphere. The surface reflectance is 
assumed to be lambertian with a single distant light source at gradient 
point p$ = 0*7 and q* * 0*3* 
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Figure 2-8 The gradient points, 
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Figure 2-9 The gradient points of figure 2-8 which receive illuolnatlon from 
the light source (i.e.. the gradients with i < n/2). 
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Figure 2-10 The restricted subsection of contour deternined for etch sampled 
image point. 
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Figure 2-11 The superposition of figure 2-9 and figure 2-10. Crosses nark the 
exact gradient, determined analytically, while each subsection of contour 
indicates how well the program has determined that gradient. 
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Z9 DISCUSSION 

The performance of the algorithm can be judged by examining the 
angular freedom reoaining in each subsection of contour C, . This anguUr 
freedom is noasured in two ways. First. since each point In gradient space 
defines a surface orientation, the naxlnun angle subtended between points 
remaining in each C, can be neasured. Define the angular spread in surface 
orientation at image point I, as: 

max|ff|cos(*) = [Pi t9 l t"» Mw tV r I J and (pi ,q, MPz-QzJcC* J 
l[pi.qi,-l]l|(P2,qi.-l]| 

Second, define the angular spread In viev angle at iaage point 3 as: 

aax((ei-C2)|tan(ei )«Vpi 2 +<n* , t*n(e? )=>/pz'+(|j ? and (pi ,qi ),<P? ,q?)<Ci ) 

To tabulate the results, the sampled image points were split into two 
classes. First, consider all sample points corresponding to object points 
within 45° view angle (i.e.. lying within gradient space circle 
p • q =1)- Second, consider all sample points corresponding to object 
points within 60° view angle (i,e., lying within gradient space circle 
p 2 4 q 7 i 3), Table 2-1 summarizes the results: 
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Points Within 


4? 


Points Within 60" 




View Annlt! 




Viow Anglo 


Surface Orientation: 








Mean Angular Spread 


8.9° 




9.9° 


Standard Deviation 


6.1" 




7.0 a 


Best Case 


0.5" 




0.5° 


Worst Case 


36.3° 




41.7° 


View Angle: 








Mean Angular Spread 


4.7° 




5.0° 


Standard Deviation 


ZA° 




2.5° 


Best Ca.se 


0.5" 




0.5° 


Worst Case 


10.9° 




n.9° 




TABLE 


Z-i 





These measures reflect total angular spread. If a choice algorithm is 
adopted which selects the "correct - answer to be at the midpoint of the 
angular spread, this choice Is guaranteed to be no noro than half the 

angular spread in error. Thus, the upper right portion of table 2-1 can be 

interpreted as follows: 

On the average, by sanpti»v at an imge spacing oj $ points 
in X and Y t the algorithm was able to position image 
points, corresponding to surface points less than 8(f* in 
viea angle, to unthin 6° of their true orientation in 
space* The standard deviation of this measure oner alt 
such points was 3.5° while the worst case point tias located 
to within 2t* oj its true orientation in space. 

The performance of the algorithm depends critically on three factors: 

the ability to hypothesize monotonici ty relations between selected image 

points, the nature of constraint propagation and the nature of the 

reflectance map. Hypothesis based mechanisns aro always faced with a 

chicken and ogg dilemma. If one has no a priori constraint on surface 

curvature, one can say nothing about ordering image points with respect to 

changes in view angle and changes to direction of steepest descent. On the 

other hand, the noro constraint on surface curvature, the more reliably one 
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con order image points with respect to changes in view angle and changes to 
direction of steepest descent. The discussion of convexity gives some 
indication of how one can use general a priori assumptions about surface 
curvature to specify monotonlcity relations- In the next chapter, 
situations are explored in which additional curvature constraint can be 
exploited to simplify inagc analysis. 

In the example above* the hypothesis mechanisn used always total ly 
ordered each of the nine pattern points. Still, the algorithm did not 
determine each gradient exactly. In some sense, it should have. For a 
solid of revolution, with axis of rotation about the viewing direction, the 
direction of steepest descent at each linage point Is given by 
= tan- '(y„/xe) where (x*.y») is a pseudo-origin as defined above. The 
algorithm propagates constraint by casting away sections or contour that 
are inconsistent with the current hypothesis criteria. If the hypothesis 
criteria are valid, the algorithn will never cast away a correct solution 
to the equation I{x.yJ = R(p.<i) . On the other hand, this does not imply 
that each point remaining on a subsection of contour is a possible solution 
to the equation I(x.y) = R(p.q) consistont with the current hypothesis 
criteria. The algorithm, as implemented, does no point by point analysis. 
Any case analysis is a risky business in a numerical algorithm which is. In 
principle, dealing with a continuous rather than discrete donwin. The 
ability to achieve a tight global constraint Is, unfortunately, related to 
the ability to achieve a tight local constraint- This, in turn, is related 
not just to the success of the hypothesis mechanism but also to the nature 
of the reflectance nap. 
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In the example, surface orientation was most poorly constrained in the 
third and fourth quadrants of gradient space. In the third quadrant, the 
reflectance nap contours locally approximate sections of circles centered 
at gradient space origin- Here, view angle is tightly constrained but the 
shape of the reflectance nap contours provides little constraint on 
direction of steepest descent. In the fourth quadrant, the reflectance Dap 
contours locally approximate radial lines emanating from the gradient space 
origin. Here, the direction of steepest descent is tightly constrained but 
the shape oT the reflectance nap contours provides little constraint on 
view angle. 

The nature of the reflectance map is determined by two factors: the 
surface photometry of the object being viewed and the light source, object 
surface and viewer geometry. In general, one would not expect to have much 
control over the surface photonetry of the objects being viewed. In 
principle, however, one is freo to vary the light source position to 
achieve optimal results with the algorithm. This idea is the basis for tho 
method of photoiietric stereo which determines surface orientation using 
multiple images taken under tho same object surface and viewer geooetry but 
with different light source positions (see Chapter 3.4). 

The notions of local surface orientation, gradient space and the 
reflectance map are important tools for image analysis. The beauty of the 
gradient space fornulation 1$ that it allows physical constraints on the 
object surface to be expressed as simple geometric constraints on the 
gradient space contour of possible solutions to the image-forming equation 
Kx.y) = R(p.q). Tho algorithm discussed in this chapter provides a simple 
mechanism for propagating these geometric constraints. 
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1 EXPLOITING ADDITIONAL CONSTRAINT 
Known properties of surface curvature- can be used to simplify image 
analysis. Earlier, it was denonstratcd how convexity and concavity 
constrain the possible matches between image points and thoir corresponding 
gradient points. Here, this analysis is extended to explore additional 
situations in which known properties of surrace curvature can be exploited 
in mage analysis. 

Horn has considered situations in which special properties of surface 
phototaetry simplify iaage analysis (Horn 75], [Horn 77]* This chapter 
presents a complementary study. Simplifications due to special properties 
of surface photometry constrain the reflectance map. When viewed from 
great distances, the material of the aaria ot the moon, for exaople, has a 
reflectance function which is constant for constant cos(l)/cos<e)« This 
results in a reflectance map whose contours of constant R(p.i|) are a family 
of parallel straight lines. Inage analysis is simplified since the base 
characteristics used to solve the nonlinear partial-differential equation 
become a predetermined family of straight lines, independent or surface 
topography. Here, an arbitrary reflectance map is allowed. Instead, 
simplifications arise from known properties of surface curvature. It is 
shown that inage analysis siopliflcs for surfaces with constant image 
Hessian, for surfaces that are singly curved and for surfaces corresponding 
to objects that are generalized cones. 

The results developed for these special cases deepen our understanding 
of the problem of cooputing object relief fron inage intensity. Relating 
the results developed here to the generalized cone representation of 
[Agin * Binford 73], [Karr 77a] and (rtarr 77b] helps to delineate shape 
information that can be determined from object boundaries and shape 
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information that can bo determined from shading. The results developed are 
of immediate practical inportance .since nany surfaces occurring in 
industrial parts design and manufacture are also constrained by the surface 
curvature properties discussed here* 

Finally , a novel technique called photometric stereo is presented. 
Photometric stereo exploits the additional constraint provided froo a 
second imago taken with the sa»e imaging geometry but with a different 
direction of incident Illumination. 

11 SURFACES WITH CONSTANT IMAGE HESSIAN 

This section examines the case for which the image Hessian matrix H 
is constant. This is useful for two reasons. First, if indeed the Hessian 
matrix H is constant over a region of the image, then local Information 
gathered about the Hessian becomes global information. This allows one to 
combine local evidence about the Hessian and determine it cooplctoly. 
tSetond, if one assumes that H is constant over a small region of tho 
image, then it is possible to use the intensity values present in that 
region to approxioate surface curvature over that region. 
In developing the basic imagc-foming equation 

I(x ( y) = R(p.q) 
an effort has been made to stress the fact that the reflectance map is not 
an image. Rather, it is a convenient representation of how surface 
orientation determines inage Intensity for a particular object material and 
object surface, light source and viewer geometry. In this section. It will 
be convenient to think of the reflectance map as an inage. 
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For the reflectance cup to be an image of some object, the surface 
must satisfy the two differential equations; 



The fatally of paraboloids: 





= X 


3y 


- y 


2 


- + fc 



(3*1-1) 



(where k is an arbitrary constant) satisfy the two differential equations. 
For those surfaces, the basic Image-forming equation becomes: 

I(x,y) . R<x,y) 

iti s : .. the reflectance nap may be thought of as the image of a paraboloid. 
As a first observation, this suggests that a paraboloid would aake an ideal 
calibration object for empirically determining a reflectance map. 
Typically, a sphere is used. A sphere serves almost as well If one Is 
willing to live with a loss of accuracy as the viewing angle increases. 

For the problem at hand, it is not so much of interest that the 
reflectance map is the image of a paraboloid, but that, for such a surface, 
the linage Hessian matrix H is constant. Indeed, for the fatally of 
paraboloids described by (3.1.1), The Hessian natrix H is the Z x 2 
identity matrix. In general, in order to have a constant Hessian H, a 
surface with explicit representation z ■ f(x a y) must have no terms of 
higher than second order (in x and y). That Is, the surface can be 
described by the form: 

z * ax 2 + by 2 ♦ Zcxy ♦ dx + ey + f (3*1*2) 

The corresponding Hessian becomes: 
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a c 
H = I 

C b 

To rule out degenerate situations, assume that the surface described by 

(3.1.2) Is fully second order. That is, assume ab - c 7 * 0. (This 

assumption inplies that H 1$ nonsingtilar. ) Subject to this assumption* 

(3,1.2) describes either an elliptic paraboloid or a hyperbolic paraboloid. 

Suppose that image point (xo.yo) is known to correspond to gradient 
point (po.qo )* In order to deternine the inage Hessian matrix II at 
(xo.yo ), it is sufficient to determine the raovenent in gradient space 
corresponding to two linearly independent movements in the imago at 
(xo.yb). Horn's method for obtaining shape fron shading information Is 
based on the observation that, if [dx.dy] ■ [Rp.fcjjds. then 
Idp.dq J ■ [I, ,I T ]ds [Horn 75 J [Horn 77 J. (See Appendix A.l for details). 
This gives one piece of inforaation about the Hessian matrix H at (xo»yo). 
Iterating this procedure provides additional pieces of information as a 
characteristic strip is expanded across the image. In general, however, 
this inforaation cannot be combined since the Hessian itself changes along 
the characteristic strip. But. if the surface is of the form (3,1*Z). then 
the information can be combined- 

For most regions of a reflectance map. the direction defined by 
[R P .Rq ] does not change very rapidly along a characteristic strip. 
Nevertheless, it does not require much change to determine H to high 
accuracy. But note a paradox. The situation in which Horn f s method 
achieves its greatest simplification (i.e.. lunar topography) is the 
situation for which the base characteristics are predetermined straight 
lines in the image. If. in fact, the base characteristics are straight 
lines then it is never possible to obtain linearly independent movements in 
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the imago by characteristic strip expansion! In such a case, it is not 
possible to say anything about how H behaves in directions other than the 
direction detained by the base characteristic, even In the cases for which 
H is constant over the whole image. Roughly speaking, the ability to 
approximate the Hessian II locally at an ioege point (xo.yo) is related to 
the curvature of the base characteristic passing through (xo,yo). If the 
base characteristic has high curvature at (xo.yo) then good conditioning, 
in the sonse of linear independence, is achieved between closely spaced 
ioago points (i.e., over areas in the image where the approximation of 
constant H is likely to be a reasonable one). 

There are two factors which act together to determine how the image 
Hessian matrix H maps a noveoent [dx,dyl in the Imago into the 
corresponding movement (dp.dq] in gradient space. First, in moving from an 
inago point (xo*yo) to an adjacent point fxi ,yi ) there is a change in local 
surface orientation due to actual surface curvature. Second, because the 
object is viewed obliquely, the actual change in local surface orientation 
it altered by the foreshortening of the surface in image projection. 
Fortunately, it is possible to decouple surface curvature from the surface 
foreshortening induced by a nonzero view angle e. 

Suppose image point (x.y) is known to correspond to the gradient point 
(p,q). Let H be the Hessian matrix H at (x.y). Define a matrix C at 
(x,y) by: 



C * cos 3 (e> 



<l*+i -PQ 
-pq p**l 
Equlvalently. one can solve (3.1.3) for II and obtain: 



H (3.1.3) 
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II ■ l/cos{e) 



(3.1.4) 



P 2 *l pq 
pq q ? *l 

The matrix C. defined by equation (3.1.3), corresponds to an 
object-centered definition of surface curvature. Let 1/n and \/r? be the 
two eigenvalues of C with corresponding eigenvectors wi and w? . Then n 
and r? are the two principal radii of curvature of the object surface at 
(x,y) with corresponding directions, in the image, given by wi and w; . 
Note that the values n and r? are independent of the imaging geometry and 
depend only on the actual surface topography at (x.y). The first two terms 
of the right-hand side of equation (3.1.4) account for the "distortion* in 
II due to the oblique viewing angle e. The matrix C is an object-centered 
definition of surface curvature while the natnx H a viewer-centered 
definition of surface curvature. If the gradient (p»q) at image point 
(x.y) is known, then equations (3.3.3) and (3.1*4) make these two 
definitions of surface curvature equivalent. 

Constraints on surface curvature are constraints on the matrix C. 
Equation (3*1*4) must be used to translate any such constraints into 
equivalent constraints on H. Some idea of the effect of surface 
foreshortening can be seen by exanining the structure of H in nore detail. 
First, the determinant of H is given as: 



dotl HI = 



1 



1 



cos*(e) nr? 
Roughly speaking. H contains a "scale factor* of I/cos ? (e) due to the 
oblique viewing angle e. Flore precisely, the product 



l/cos(e) 



p*+i pq 

pq q'*l 
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has eigenvalue l/cos 3 {eJ in the direction of steepest decent and eigenvalue 

l/cos(e) in the direction of the contour of constant e. 

Elliptic and hyperbolic paraboloids are the unique surfaces for which 

the "increase" in H due to an increasing view angle e is precisely offset 
by the decrease in 1/n and 1/r? due to the changing surface curvature. 

3.1,1 APPROXIMATING THE IMAGE HESSIAN LOCALLY 

Let us see how the assumption that H is constant over a small region 
oT the inage can be used to approximate the inage Hessian at an inage point 
(x.y) known to correspond to the gradient point (p,q). The basic 
observation used here Is that multiplication by a constant H defines a 
one-to-one continuous mapping between & circle in the image centered at 
(x.y) and an ellipse in gradient space centered at (p»q) (see 
Appendix A. 3)* If two linearly independent directions Idxi.dyi] and 
[dx;,dy2] and the corresponding [dpi.dqi] and [dp? t dqz] are known, then H 
is detcroinod by: 



One 



II 



correspondence 





dpi dp; 




dx] dx? 




dqi dq? 




dyi dy? 


c 


an be 




tied 



(3.1.5) 



down 



immediately. 



ir 



[dx.dy] = [R 0l R,lds then tdp.dq] ■ [I, ,1, Ids. Thus. (3.1.5) can be 
rewritten as; 



II = 



I. dp 




«» dx 


I, dq 




P« dy 



-I 



(3.1-6) 



Equation (3,1-6) still admits an infinite nuober of solutions. 
However, for any particular second choice [dx.dy], linearly independent 
from [Rp,RqL the geometric constraints developed in Chapter Z can be used 
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to constrain the set of [dp f dqj that can correspond to the given [dx t dy]. 
Each Jdp.dq] not eliminated by geometric constraint defines a possible 
solution to (3. 1.6), For each possible solution, one can proceed to sample 
the intensity values along the inape circle centered at ( =.y) and of radius 



ds = vdx + dy^ • For each point on this circle, the corresponding 
reflectance value R(p*dp ( q*dq), where [dp,dq] T = H [rk.dy] 7 , can also be 
sampled. The "best fit" Hessian at (x,y) is defined to be that choice of 
(dp.dq] which nmimizes the least square error between the stapled 
intensity values and the corresponding sampled reflectance values. 

Consider an example. Suppose image point Po c (xo»yo) is known to 
correspond to gradient point (po.qo) on a section of surface assumed to be 
convex. Then, for a particular choice of ds, the lmape Is sampled at the 
four points: 

?2 ■ (XOt»+ds) 



P3 = (xo-ds,yo 



Pi * (xo+ds.yo) 



Po » (*o,Yo) 



P4 ■ (xo,yo-ds) 
From the four intensity values thus obtained, the four reflectance map 
contours, corresponding to the four image points Pi, P?, P3 and \\ t . are 
determined. (Mote: If bounds on the eigenvalues of H are already known, 
then the range in gradient space used in the initial determination of the 
four reflectance map contours can be tightly constrained.) Figure 3*1 
Illustrates the four contours obtained for a particular choice of (xo,yo), 
(po,qo) and ds. using the laobartian sphere example of Chapter 2.6. No a 
priori constraint is assuaed on the magnitude of the eigenvalues of H. 
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Convexity allows us to assert the two inequalities: 

dpdx + dqdy z 

After applying these two inequalities to each of the four contours of 
figure 3-1, the constrained contours of figure 3-2 are obtained, 

Thore is one additional constraint available. H defines a linear 
transformation. For tho particular choice of Pi , Pz, Pj and Pa above, 
observe that the [dx,dy] used to Dove from <xo,yo) to Pi is the negative of 
the ldx t dy] used to nove fron (xo,Yo) to P3. (Sinilarly, for P2 and Pa.) 
Thus the corresponding [dp,dQ]'s oust be the negative of each other- In 
particular, the nagnitude of the [dp,dq]'s must be equal and they oust be n 
radians out of phase. Each point in the contour of possible (p.q) for Pj 
which does not have a watch in P3 whose distance fron (po.qo) is the sane 
and whose angular position with respect to (po»qo) is n radians out of 
phase can be excluded. Similarly, each point, in the contour of possible 
(P.q) for P3 which does not have a match in Pi whose distance fro» (po^qo) 
is the same and whose angular position with respect to (po»qo) is ii radians 
out of phase can be excluded. (Note: these criteria must be applied with 
caution otherwise all points will be excluded due to the fact that H is 
not really constant.) The identical constraint holds between P* and P* , 
Figure 3-3 shows the contour remaining when the above criteria are applied 
to oxclude "obvious'* bad matches. 

Of the two pairs of contour, corresponding to Pi and P3 and to P? and 
P4. select the pair that is closest to being */2 radians out of phase with 
the direction [I*»Iy]< (This choice achieves superior conditioning in tho 
estimation of II . ) For each [dp ( dq] remaining in the most highly 
constrained contour from the selected pair, use (3,1.6) to estUata H. 
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Figure 3-4 shows the family of ellipses generated by each point remaining 
on the contour for P*. The vector superimposed on figure 3-4 points in the 
direction [I..I y ]. Figure 3-5 shows least squares "boat fit" Hessian 
matrix at (xo.yo). 

The "best fit" Hessian matrix of figure 3-5 has done a good job of 
solving for the curvature of Lhe object surface at (xo#yo). It is 
Important, however, to be precise about what it is that has been computed* 
Thorn are two assumptions underlying tho computation. First, it is assumed 
that the Hessian matrix H is constant over the area in the image 
determined by the circle centered at (xo.yo) and of radius ds. Second, It 
is assumed that the gradient point (po.Qo) corresponding to image point 
(xo.yo) is known. Equation (3.1.3) tells how to choose ds to compensate 
for the foreshortening due to an oblique viewing angle e and thus achieve 
uniform sampling resolution over the entire surface. Actually, by sampling 
equally spaced points around the image circle, different weights are 
applied to different directions along the surface. A more sophisticated 
sampling scheme is required to remove this difficulty. A more serious 
restriction, however, is related to the second assumption. If the center 
of the ellipse has not been accurately positioned then a least squares 
"best fit 1 * is akin to groping around in the dark. This example illustrates 
not so much a useful method for finding the gradient points corresponding 
to a set of image points but the fact that tho Intensity values in an image 
can be used locally to constrain the behavior of the image Hessian matrix 
H in directions other than those defined by the current [Ii.Iy]* 
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Figure 3-2 The restricted 
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Figure 3-3 The restricted subsection of contour for Pi, Pj f P 3 
obtained when "linearity* is used to exclude obvious bad patches. 



and P* 



R, .1. Uuuiliiam 



77 



Exploiting Additional Conitralnt 



i 



n-i 



mT 



"oT 



"^. 



^r 



OJ 



Fieure 3-4 The family of ellipses which characterize the uncertainty 
remaining In the detcroination the image Hessian H. The vector at Po is In 
the direction of the noraal to the contour of constant Intensity In the 

at (xo,yo) 
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Figure 3-5 The ellipso corresponding to tho "best fit' Image Hessian at 
(xo.yo). 
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- 
3? SINGLY CURVED SURFACES 

An lnporiant class of surfaces in differential geometry arc those 
which have tho property that, through every point on the surface, there 
passe* at least one straight line lying entirely on it. Such a surface Is 
called a ruled surface. The straight line lying entirely on tho surface is 
called a rating. If a ruled surface has the additional property that all 
points on a given ruling have the sane tangent plane, then the surface is 
called a developable surface (or a torae). [Huffman 75] charmingly refers 
to developable surfaces as "paper* surfaces and proposes that such surfaces 
possess a complexity that is midway botween that of a completely general 
surface and that of a plane surface. ([Huffman 75] also presents an 
interesting characterization of properties of "paper - surfaces In terms of 
Gaussian curvature and the Gaussian sphere.) 

More, the notion oT a singly curved surface is developed in terms of 

properties of the Hessian matrix H* Singly curved surfaces are shown to be 

equivalent to developable surfaces (except that a planar surface, while 

certainly developable, will not be considered singly curved). Finally, the 

Image analysis of singly curved surfaces is discussed. The result will 

denonstrate that, from an inage analysis standpoint, singly curved surfaces 

do indeed possess a complexity that is between that of a completely general 

surface and that of a plane surface. 

Definhioii. Let z = f(x,y) describe a snooth surface and let 
Xi and X? be the two eigenvalues of the corresponding 
Hessian matrix If at an inage point (xo.yol* The surface 
is said to be singly curved at fxo.ltoJ if and only if 
exactly one of Ai and >2 is equal to zero. The surface is 
said to be singly curved if it is singly curved at each 
image point (x,y) on tho surface. 
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This viewer-centered definition of a singly curved surface can be 
related to the object-centered principal radii or curvature. From 
equations (3.1.3) and (3.1.4) above, it can be shown that one of the 
eigenvalues \\ and > 2 or H is zero IT and only if one of the principal 
radii of curvature n and rz is infinite (see Appendix A. 5). 

For surfaces expressed in the form z = f(x,y), the equation: 



alrUxXi a»r(x.vi « [ a ? f(x.y) j 



m 



a*y 



(3,2>1) 



dxdy 



is the differential equation characterizing developable surfaces. In the 
current notation, this equation is equivalent to the equation: 

detllll = (3.2.2) 

Equation (3.2.2) is satisfied at an image point (x,y) if and only if at 
least one of the eigenvalues of the corresponding H is zero at (XiY)> 
Thus. (3.2.1) is satisfied for all surface points (x,y) if and only if at 
least one of the eigenvalues of H is zero at each image point (x,y). If 
both eigenvalues of H are zero for all image points (x.y), then the 
corresponding equation z = f(x,y) describes a plane. For convenience. 

assume that at least one of Xj and X* U nonzero at each laage point (x.y). 

Thus, a nonplanar surface z = f(x.y) is developable if and only if it is 

singly curved* 

Let z = f(x.y) be a singly curved surface. Suppose that a point 
(xo.yo) in the inage is known to correspond to a point (po,qo) in gradient 
space. Then, the Hessian matrix H at (xo.yo) is completely determined. 
Indeed, II is given as the natrix product: 



11 = 



cos(o) -sin(a) 
sin(a) cos(a) 



X 




cos(a) sin(cO| 
<sin(a) cos(a). 



(3.2.3) 
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where X = /l, 2 ♦ U 2 

R P cos{a) * RqSin(a) 



and t»r tf ) * I 



tan(cr) 

la 

As before, I* and I v denote the first partial derivatives or I(x,y) at 

inage point (xo,yo) and R* and R* denote the first partial derivatives of 

R(p.q) at the corresponding gradient point (po*<to>* 

Thus, given any initial inage point (xo>yo) known to correspond to 

gradient point (po.qo). the Hessian matrix of z * f(x,y) at (xo-Yo) is 

determined by (3*2*3)* The Hessian matrix 11 so determined can then be 

used to find the new gradient corresponding to an arbitrary movement 

[dx.dyj in the image according to the equation: 

[dp.dq] T - H [dx.dy] T (3.2.4) 

The operations enbodUd in (3,2,3) and (3.2*4) above can be iterated 

to trace out an arbitrary family of curves on the surface. For singly 

curved surfaces, one is not conTined to tracing out the characteristics of 

Horn's original method for obtaining shape from shading information 

[Horn 75], [Horn 77). 

This result should not be terribly surprising. The fact that H has 

one zero eigenvalue neans that there is one direction of movement in the 

image which results in no change lo surface orientation. The orthogonal 

direction a is determined by the vector [I R ,i v ]- The component or any 

movement [dx.dy] perpendicular to [ I* , I v ] ia in the direction of a ruling 

on the developable surface z = f(x,y} and thus does not cause a change to 

the gradient (p,q). The component of [dx.dyl in the direction [1*,I V ] 

causes a change [dp,dq] to the gradient in the direction a where the "scale 

factor" for that change is given by the value of *. 
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The points in gradient space corresponding to points on an arbitrary 
singly curved surface z c f(x.y) are constrained to lie on a one-parameter 
curve in gradient space. This is just another manifestation of the 
observation that singly curved surfaces possess a complexity midway between 
that of a plane surface, where surface points map into a single point in 
gradient space, and that of a completely general surface, where surface 
points nap into a two-parameter region in gradient space. 

Figure 3-6 is the image of a right circular cone of base radius b and 
height h generated using the reflectance map of figure 2-4, (For this 
exaople. h » Zb,) It can be shown that the points in gradient space 
corresponding to points on a right circular cone lie on the one-parameter 
curve in gradient space given parametrically by: 

P ■ Unit) (3.2.5) 

Q » Z~ k 777 (3.2-6) 

h cos(t) 

where -n/2 < t < */2. The paraaeter t h-as a physical interpretation, Tho 
circular cross-section of the cone can be represented. In cylindrical 
coordinates, by the function p{9) = I, Were $ measures angular position 
about the y~axis. If is chosen so that - - points in the direction of 
the viewer, then the parameter t in (3.2.5) and (3*2.6) is this angle tf. 

Figure 3-7 shows the curve in gradient space determined by the 
parametric equations (3.2,5) and (3.2.6) superimposed on the reflectance 
map used to generate the image of figure 3-6. There exists a one-to-one 
continuous mapping between any horizontal image intensity profile froo 
figure 3-6 and the curve in gradient space determined by (3.2.5) and 
(3.2.6). Thus, finding the point (po.qo) In gradient space corresponding 
to any image intensity point I(xo.yo) * a fro* figure 3-6 simplifies to the 
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problcn nf deteraining reflectance nap values on the curve given by (3*2*5) 
and (3.2.6) for which R{p,q) = a. If an intensity profile is scanned 
sequentially from left to right then possible multiple solutions can be 
resolved by choslng the solution which is "closest" to the previous 
solution in the direction of increasing t. This follows as a consequence of 
the general fact that all 1*1 continuous iiappings are raonotonic and the 
particular observation, for this case, that the mapping Is monotonically 
nondecreasing. 

3.3 GENERALIZED CONES 

The previous section considered the Imaging of singly curved surfaces. 
This section extends that work to consider a more general class of surfaces 
called generalized cones. Generalised cones are doubly curved. The 
curvature of a generalized cone, however, conveniently decouples, in an 
object-centered representation of shape. For appropriate viewing 
conditions, this decoupling carries over to images of generalized cones. 
These images can he analyzed •alnost" as if the surface were singly curved. 
The first result of this section is to denonstrate that, if the cone axis 
is parallel to the viewing plane, then images of right generalized cones 
with circular cross^section can be analyzed exactly as if the surface were 
singly curved. In this case, the additional information required to 
account for the second degree of freedom in curvature is not embedded in 
the intensity values present in the image but rather in the objoct 
silhouette. The more general case, in which the cone axis makes a nonzero 
angle with the viewing plane, is ftoro difficult to analyze. The sane 
decoupling of curvature holds but this decoupling is more difficult to 
recover in an Image. Finally, a few results are given for the case in 
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Figure 3-6 The synthesized Image of a right circular cone, with height equal 
to twice the base radius and axis parallel to the image plane. The surface 
reflectance is laabertian with a .single distant light source at gradient 
point p$ = 0.7 and q* = 0,3, 
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Figure 3-7 The one parameter curve in gradient space corresponding to points 
on the cone of figure 3-6. 
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which the cross-section is allowed to bo an arbitrary convex function- 

This section is important (or two reasons. First. It extends the 
approach taken In the previous section to a broader class of surfaces . 
Second. It is an initial attempt to construct a bridge between methods of 
determining surface topography by analysing the intensity variation across 
smooth sections of object surface and methods for determining object shape 
by analyzing the occluding contours present in an image. 

This section is preliminary. The question asked is. "If the object is 
a generalized cone, then how does this constrain the intensity values 
recorded in an image of the object? - The question ono really wants to ask 
is, "How can the intensity values recorded in an image be used to determine 
whether the object Is a generalized cone and. if so. to determine its axis 
and cross-section function?" At best, the results here correspond to a 
method for checking the consistency of hypotheses about surface shape 
determined from an analysis of the object silhouette against the actual 
intensity values recorded over smooth sections of the object surface. 

The concept of a generalized cone has its genesis in the generalized 
cylinder representation of [Agin & Blnford 73], There, generalized 

cylinders were used as convenient representation scheme for describing 
complex shapes. Generalized cones arise in the work of [Harr 77a], 
[Harr 77b], Here, the generalized cone emerges not so nuch as a convenient 
representation schene but rather as an interpretation that is forced if one 
tries to develop a theory of how to infer the shape of objects from their 
silhouettes. 

A gencrotized cone Is defined to be the surface swept out by moving a 
simple smooth cross-section p(()J along a straight axis A, at the same time 
nagnifying or contracting it in a sooothly varying way. Let h(X) be the 
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axial scaling function whore A denotes distance along the A axis. The 
angle 4 between the axis A and a plane containing a cross-section is 
cal led the eccentricity of the cone. For the present, it will be 
convenient to add two additional simplifying assumptions* First, assume 
thdi the eccentricity 4 = */2. With this first assuaption. the cone is 
called d right generalized co«e. Second, assunc that the cross-section is 
circular. Without loss of generality, one can further assume that the axis 
A passes through the center of the circle. With this second assumption, 
the cone is called a right generalized cone with circular cross-section . 

[flarr 77al has defined methods for finding the projection or the axis 
A on the image plane of a generalized cone from an analysis of its 
occluding contour. The goal here is to provide a complementary study to 
interpret the Intensity values recorded from the interior smooth sections 
of a generalized cone. 

Let us begin with one further siopiification. Assume that the axis A 
is parallel to the imago plane- Since a rotation of object space induces 
on equal rotation in gradient space, one can. without loss of generality, 
assume that the axis A coincides with the image y-axis. This lost 
assumption about the viewing diruction will allow for the convenient 
decoupling of the generalized cone's curvature so that the mage analysis 
problem boconcs equivalent to that of analyzing a singly curved object. 
Distance along the axis A is equal to distance along the image y-axls so 
that the axial scaling function can be denoted as h(y). Let the circular 
cross-section function be denoted by cl9) c I. where $ = points in the 
direction of the viewer. It can be shown that the points In gradient space 
corresponding to points on such a right generalized cone with circular 
cross-section lie in the two-parameter region in gradient space given 
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parametrically by; 

P ■ tanCtf) (3.3.1) 

Q = ^**1 (3.3.2) 

cos(tf) 

where h'(y) denotes the derivative of h(y) with respect to y and B varies 
between -n/2 and n/2. 

Figure 3-6 was an example of a right generalized cone with circular 
cross-section, A right circular cone of base radius b and height h has 
axial scaling function h(y) where: 

h'(y) = "* 
h 

In general. (3.3.1) and (3.3*2) define a two parameter region in gradient 

space. One of the paraneters is 9 and the other is h'(y). Of course, if 

h'(y) is constant, as in figure 3-6, then the surface is singly curved. 

The interesting observation, however, is that when the axis A is parallel 

to the viewing plane, the value of h'{*) can always be determined directly 

from the boundary contour. For a particular value of y. the curve In 

gradient space generated by (3.3.1) and (3.3.2) is a scaled version of the 

curve illustrated in figure 3-7* Differing values of -h'(y) introduce a 

different scale factor In q. Thus, finding the point (po,Qo) In gradient 

space corresponding to an laage intensity point l(xo.yo) = simplifies to 

a two step process. First, for the particular value of yo, determine 

h'(yo) as the rate of change of object radius with respect to y, or 

equivalently as one half the rate of change of object diameter with respect 

to y, at the object boundary points along the image profile y = yo . Second, 

as in the case of a singly curved object, scan a horizontal intensity 
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profile to determine the correct reflectance value on the curve given &y 
(3*3.1) and (3.3.2). 

Figure 3-B is a more general exanple. Hero, the axial scaling 
function is a sinusoid while the cross-section function remains circular. 
The surface depicted in flflure 3-B is doubly curved- Yet, the curvature 
decouples so that, fron a i»age analysis point of view, the surface behaves 
as if it were singly curved. Figure 3-9 superimposes a collection of these 
curves on the reflectance map used to generate figure 3-8. (Note: the 
sphere example of Chapter 2.8 is also a right generalized cone with 
circular cross-section with an axis A that can always be chosen parallel 
to the viewing plane!) 

Now consider the case in which the axis A of a right generalized cone 
with circular cross-section is not parallel to the viewing plane. Once 
again, a rotation of object space induces an equal rotation In gradient 
space so that, without loss of generality, assume that the projection of 
the cone's axis A on the image plane coincides with the image y-axis. Let 
the angle between the A axis and the viewing plane be * (measured so that 
positive 6 implies that the A axis is tilted towards the viewer). In this 
situation, distance along the image y-axis is sioply a foreshortened 
version of distance along A. In particular, 

y = > cos(*) 
so that the derivative of the axial scaling function h(X) is given by: 

h'(M = cos<*) h'(y) = cos<*) rat« of change of image diameter 

Now. tilting object space an angle $ about the x-axis takes gradient point 
(p-q) onto gradient point (p, ',<!') where: 
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Figure 3-8 Tho synthesized Image of a right generalized cone with circular 
cross-section and axis A parallel to the Image piano. The axial scaling 
(unction h<y) is a sinusoid. The surface reflectance Is 1 amber U an with a 
single distant light source at gradient point p, = 0.7 and q, = 0.3. 
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Figure 3-9 The region in gradient space corresponding to points on the right 
generalized cone of figure 3-8. plotted as a fanily of curves. Note that 
the region in gradient space lies beloM the curve determined by the minimum 
value of h'(y) and above the curve determined by the Maximum value of 
h'(y). 
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P 1 



cos(#) ♦ q sin(*) 



q . , - slnt+) - g cos<«fc) 
cost*) + a sin(#) 

The points in gradient space corresponding to points on a right generalized 

cono with circular cross-section lie in the two-parameter region In 

gradient space given paranetrically by: 



P ■ ^^ (3.3.3) 

tos(*)[cos(0) - sin(*)h'(y)] 

q = -tan(#) ♦ ±Hti (3.3,4) 

cos(*)tcos(tf) - sin(*)h'(y)] 

where h'(y} denotes the derivative of h(y) with respect to y and 
-rt/2 < S < n/Z. For a particular value or y, the points corresponding to a 
cross-section on a right generalized cone with circular cross-section again 
lie on a ono-pararaeter curve in gradient space. If # is known, then one 
can proceed as before, with only a slight complication in the mathematical 
expressions required. In general, however, the value of £ is unknown. 
This leads to two difficulties. First, without knowing *, the curve in 
gradient space generated by (3.3.3 ) and (3.3.4) cannot be determined. 
Second, without knowing £. the intensity profile in the image corresponding 
to this curve cannot be determined. 

Consider a fixed value of y = yo. Let (xi.yo) denote the "left" 
boundary point and (xp.yo) denote the *rlght* boundary point determined by 
the intersection of the line y a yo with the inage silhouette. By the way 
the image axes have been aligned with respect to the cone, it must be the 
case that x, = -x* . Now, the image intensity profile corresponding to the 
curve generated by (3.3.3) and (3.3.4) lies on the ellipse centered at 
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(O.y.j) with major axis x* , parallel to the imaae x-axis» and minor axis 
|xp sln(£)|, parallel to the ieage y-axls. If # > (i.e., the cone is 
tilted toward the viewer), then the correct profile corresponds to the 
lower half or the ellipse. If $ < (i.e., the cone is tilted away from 
the viewer), then the correct profile corresponds to the upper half of the 
ellipse. 

Thus, for any hypothesized value of #. it is possible to determine a 
faaily of profiles In loagc space and the corresponding family of curves in 
gradient space. The gradient corresponding to each inage point can then bo 
determined once again as IT the surface were singly curved. The gradients 
so determined can be integrated to determine a range profile across each 
image profile. These range profiles can bo analyzed to sec how well they 
correspond to a circular cross-section, tilted by an angle . In this way, 
an analysis of inage intensity can be used to verify the choice of 6. A 
"host fit" 6 can be determined in much the sane fashion as the best-fit 
inage Hessian was found in Chapter 3.1 above. That is, one can find the 
value of * which is most consistent with the hypothesis that the image 
corresponds to a right generalized con« with circular cross-section. 

Finally, consider the case for which the cross-section is allowed to 
be an arbitrary smooth convex function. This result to be established here 
is essential l\ a negative one. Although the curvature of a right 
generalized cone decouples in its object-centered representation, the 
assunption of an arbitrary convex cross-section function no longer permits 
that decoupling to be determined froo an analysis of the object silhouette. 
Roughly speaking, it becooes impossible to resolve the trade-off between 
axis tilt and cross-section shape* This, in turn, does not allow for the 
sinplif ication in inage analysis that is achieved whon the cross-section 
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function 15 known a priori. 

Let the cross-section function be given, in cylindrical coordinates, 
by r * pltf). To avoid unecessary complication, this discussion will be 
restricted 10 the case In which the cone's axis A is parallel to the 
viewing plane. Once again, let the smoothly varying axial scaling function 
bo h(>). It can be shown that the point* In gradient space corresponding 
to points on such a right generalized cone lie in the two-parameter region 
In gradient space given pararactrlcally by: 

p = J(#)slnf<) - fl'(l)coifl) (a 3 5J 

o(9)cos(») * (,'(»)sin|») 

q ■ ■*'<*) Bill {336) 

p(0)cos(tf) . a'(»)sint«) 

where h'(X) denotes the derivative of h(M with respect to 1 and p'(ff) 
denotes the derivative or o{0> with respect to 6. 

(3.3.5) and (3.3.6) define the two-parameter region In gradient space 
in the parameters 6 and h'(J). The decoupling between these two 
object-centered parameters remains. For a fixed value of h'(X), (3.3.5) 
and (3.3.6) determine a one-dimensional curve in gradient space. Differing 
values of h'(A) again simply scale this curve in q. Unfortunately, Tor an 
arbitrary convex cross-section function «(*), it Is not possible to 
conveniently discover this decoupling from the Image silhouette. It is no 
longer necessarily true that the "left- occluding contour corresponds to 
* = -«/Z and the ■right" occluding contour corresponds to t ■ n/Z. Thus, 
oven for the case in which A lies parallel to the viewing plane, h'(X) can 
no longer be determined directly from the silhouette. 
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If the left occluding contour corresponds to the value Bi (where 
-n < Si < 0) and the right occluding contour corresponds to the value Or 
(where < fft < n). Then, h*(M is given by: 

kifii s rate of change of object dianeter 
[sin(-* L ) ♦ iin{#«)] 

The information required to determine h'(X) is still contained in the 

silhouette but it can be used only if the values it and $r arc known. The 

values of ft and flp, in turn, depend on p!0). 

3.4 PHOTOMETRIC STEREO 

This section develops a useful alternative to the previous methods for 
determining the surface orientation corresponding to a given image point* 
The equation l(x,y) = R(p.q) is one equation in the two unknowns p end q. 
The theoretical machinery developed in this report has been oriented 
towards seeking ways to exploit additional curvature constraint in order to 
solve this underdetermined equation * Another idea is to get more 
equations. 

Traditional stereo techniques determine range by relating two Images 
of an object viewed from different directions. If the correspondence 
between picture elements is known, then distance to the object can be 
calculated by triangulation. Unfortunately, it Is difficult to determine 
this correspondence. The idea of photometric stereo is to vary the 
direction of the incident illuninatlon between successive views while 
holding the viewing direction constant. This provides enough information 
to determine surface orientation at each picture element, as will now be 
shown. 
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Suppose two mages I|(x.y) and I?(x ( y) are obtained by varying the 
direction or incident illumination. Since there has been no change in the 
imaging geooetry. each picture eleaent (x,y) in the two iaages corresponds 
to the same object point and hence to the same gradient (p,q). The effect 
of varying the direction of incident illunination is to change the 
reflectance map R(p,q) that characterize! the imaging situation. 

Let the reflectance maps corresponding to 1| (x*y) and I?(x t y) be 
Ri(p.Q) end R?(p f q) respectively. There are now two independent equations 
in the two unknowns p and q: 



(3. 4. J) 



. IiU.y) = Ri(p.q) 
I?(x.y) a R?(p,q) 

These two nonl incur equations wil 1 have at most a finite number or 
solutions. 

Two reflectance naps are requ ircd. But, further slnpllf ication Is 
obtained if the phase angle g is the sane in each view. Constant phase 
angle g is achieved when the illumination is rotated about the viewing 
direction. In this case, the two reflectance maps are rotations of each 
other. Suppose two identical distant point sources are located 
respectively at gradient (pi ,qi 1 and (p2,q;)» with corresponding 
reflectance maps Ri(p,q) and Rztp.q), so that 



-An 2 + qi* = Vpj^ 



q?' 



Let 8 be the angle of rotation that takes (pi.qi) to <P2 t qz)- Tben, 

RztP.q) * Ri<P\<T) 



whore 



cos(fl) -sin(ff) 
sin(S) cost*) 
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For example, if $ = 90*. thun the two equations (3.4,1) become: 

I|{x.y) = Ri(p.q) 
I*(x,y) = Ki(-q.p) 
There are three ways to proceeed* If image point (x,y) is known to 
correspond to the gradient point (p.q) and if [Ri» P Riq] and [R2p,R?q] are 
linearly independent, then the inage Hessian matrix H Is uniquely 
determined as: 



H 



lu U. 



I J v '?v 



Rip R20 
R lq R? q 



-1 

(3-4-2) 



(Here, the first subscript identifies the image and the second subscript 
denotes partial differentiation*) One possibility is to start at an image 
point (xo»yo) known to correspond to gradient point (po.qo) and expand a 
complete solution over a section of smooth surface using the equation 

[dp.dQ) T = H [dx.dy] T 
At each step, (3.4*2) determines H completely. Thus, the direction of 
movement [dx f dy] can be freely chosen. 

A second possibility Is to explicitly determine the finite number of 
solutions to (3,4.1). For the special case of the material of the maria of 
the noon, equations (3.4,1) are linear in p and q. In this case, (3.4.1) 
determine a unique surface orientation at each inage point, provided the 
directions of incident illumination are not collinear. In general, more 
than one solution is possible. These solutions, however, ^rt typically 
widely spaced in gradient space. They provide ideal input for the 
relaxation algorithm presented in Chapter Z. The application of simple 
curvature assumptions, such as convexity or concavity, con be used to 
quickly nail down a unique surface interpretation- 
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A third possibility is to use additional light sources to 
ovcrdeterraine the solution. Suppose the object is viewed a third tine with 
the light source placed at gradient point (p3 f q3). Call the corresponding 
reriectancc map Rstp.q). A third image hlx.y) is obtained which satisfies 
the equation: 

l3(x,y) ■ Ratp.q) 
The sot of equations: 

IiU.y) « Ri(p.q) 

I?(x,y) a R?(p,q) 

hCx.y) o R3{p t q) 
Is now overdetcrmincd. and. barring degeneracy, will have a unique 
solution. 

The images required for photome trie stereo can be obtained by 
explicitly raoving a single light source, by using multiple light sources 
calibrated with respect to each other or by rotating the object surface and 
imaging hardware together to simulate the effect of moving a single light 
source The equivalent of photometric stereo can also be achieved In a 
single view by using multiple illuminations which can be separated by 
color* 

Photometric stereo is fast* It has been developed as a practical 
scheme for environments in which the nature and position of the incident 
illumination is known or can be controlled- Initial computation Is 
required to determine the reflectance map for a particular experimental 
situation* Once calibrated, however, photometric stereo can be reduced to 
simple table lookup and/or search operations. 
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Photometric stereo can bo used in two ways. First, photonetric stereo 
is a general technique for deteraining surracc orientation at each ioage 
point. For a given imago point (x,y), the equations characterizing each 
image can be combined to determine the corresponding gradient (p<q). 

Second, photometric stereo is a general technique for determining 
object points that have a particular surface orientation. This use of 
photometric stereo corresponds to interpreting the basic inage-foraing 
equation I(x»y) * R(p.q) as one equation in the unknowns x and y. For a 
given gradient (p.qh the equations characterizing each ioage can be 
combined to determine corresponding object points (x,y). 

This latter use of photometric stereo is appropriate for the so called 
industrial bin-of-parts problem. The location in an image of "key" object 
points is often sufficient to deternine the position and orientation of a 
known object tossed onto a table or conveyor belt* 

A particularly useful special case concerns object points whose 
surTacc normal directly faces the viewer (i.e. object points with p a and 
q = 0). Such points form a unique class of image points whoso intensity 
value is invariant under rotation of the incident illumination about the 
viewing direction. Object points with surface noraal directly facing the 
viewer can be located without explicitly deternining the reflectance map 
R(p.q). Whatever the nature of the function R(p,q). the value of R(0.0) is 
not changed by varying the direction of incident illumination, provided 
only that the phase angle g is held constant. 

These applications of photometric stereo are illustrated using the 
simple, synthesized sphere example of Chapter 2.8. 
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M.I DETERMINING THE SURFACE ORIENTATION AT AN OBJECT POINT 
Suppose throe imago* Mx,y). U[x t y) and i3(x,y) are taken under a 
Tixed object surfaco viewor geometry but with a varying light source 
position. Suppose the corresponding reflectance maps are Ri(p.q), R?(p*q) 
and Ro(p.q). Choose a particular image point (xo.yo) and suppose that the 
intensities at (xo.yo) in the three images are given by litxo.yo) ■ ai , 
l2(xo.yo) = oz and l3<xo,yo) ■ 03* One plots, in gradient space, the three 
contours Ki(p,q) * «i , R?(p,q) = o^ and R3(P.q) =03- Any gradient point 
(p,q) lying on all three contours is a possible gradient corresponding to 
tha iraage point (xo,yo). If there is only one such (p.qh then this (p,q) 
uniquely determines the local surface orientation at the object points 
corresponding to the given image point (xo.yoh 

Figure 3*10 illustrates, The three images are obtained by varying the 
position of the light source. Let the first be pi = 0.7 and qi ■ 0.3, as 
in Chapter 2.8. Let the second and third correspond to rotations of the 
light source about the viewing direction of -120' and + 120* respectively 
(i.e., P2 i -0*610, qz * 0,456 and pa = -0.090. qu = -0.756). The phase 
angle g is constant in each case. Consider Image point x * 15, y = 20. 
Hero, Ii(x,y) a 0.942, Ii(x.y) a 0.723 and l3(x,y) * 0.505. Figure 3-10 
shows the reflectance map contours Ri(p.q) = 0*942, R?(p,q) ■ 0.723 and 
Rr«<D.q) = 0.505. The point p ■ 0.275. q = 0.367 at which these three 
contours intersect determines the gradient corresponding to image point 
x ■ 15, y ■ 20. 
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R 2 ( P ,q) = 0-7Z3 



Rj<p,q) = 0-34Z 




R 3 (p,q) - 0,505 




Figure 3-10 Determining the surface orientation at a given image point 
(x»y). Three superimposed reflectance map contours are intersected whero 
each contour corresponds to an intensity value at (x.y) obtained froo three 
separate images, taken under the sane inaging geometry but with dirrerent 
light source position. 
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34.2 DETERMINING OBJECT POINTS WITH GIVEN SURFACE ORIENTATION 

Suppose three mages h(x.y), l2(x.y) and lafx.y) are taken under a 
f lxod object surface viewer goonotry but with a varying light source 
position. Suppose the corresponding reflectance maps are Ri(p,q), R?(p,q) 
and Ro(p.q). Choose a particular gradient point (po.qo) and suppose that 
the reflectance values at (po.qo) in the three reflectance maps are given 
by Ri (po,qo) =ai» RzIpo^Qo) =02 and Rj(po,qo) > 03 - One plots, in Image 
space, the three contours Ii(x,y) * 01 1 I*(x.y} » a* and ta{x,y) = 03. 
There may be zero, a finite number or an infinite number of image points 
<x,y) lying on all three contours. If there is no such (x f y). then there 
is no object point in view with surface orientation given by the gradient 
( po »qo ) . If there are a finite number of such (x,y) ( then, barring 
degeneracy, each (x.y) corresponds to an object point with local surface 

orientation given by the gradient (po.qo). If the surface is not doubly 
curved, then the set of such (x,y) nay be infinite (i.e., the surface aay 
have surface orientation Given by gradient (po.qo) over a curve or region 
in the image). 

Figure 3-11 illustrates* Consider gradient point p = 0.5. q = 0.5. 
Here. Ri<p,q> - 0.974. R*(p.q> = 0.600 and Ralp.q) * 0.375. Figure 3-11 
shows the image contours Ii(x,y) = 0.974. I?lx»y) & 0.600 and 
l3{x,y) = 0.375. The point X = 24.5. y » 24.5 at which these three contours 
intersect determines an object point whose gradient is p & 0.5. q 1 0.5. 

Figure 3-12 shows how to use this technique to deterainc a 
pseudo-origin (i.e., an ioage point corresponding to an object point with 
gradient p = and q = 0). Here. Ri(p,q) = R?(p,q) = R3(p»q> = 0.796. 
Object points with surface nornal directly facing the viewer forn a unique 
class of points whose ioage intensity is invariant for rotations of the 
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light source about the viewing direction* The point x = 0. y « at which 
these threo contours intersect determines an object point with surface 
normal directly facing the viewer. This result holds even If the form of 
R(p,q) is unknown* 

3.4-3 USING PHOTOMETRIC STEREO 

With any stereo technique, there Is some trade-off to acknowledge. In 
photometric stereo, choosing a largo phase angle g allows one to achieve a 
better "conditioning" of the gradient contours that oust be intersected 
from each image. At the same tine,, a large phase angle g puts more of 
gradient space into the shadow region of one or nore of the sources. In 
the example below, a four-source scheae is considered. Each source 15 a 
99* rotation fro* its, two neighbors *Hft tbt first source again positioned 
at ps ' 0.7, qs ~ 0.3. With this configuration, all object points within 
61.7° view angle arc Illuminated by at least three independent sources. 
Further, assuning lanbertian reflectance, contours in sections of gradient 
space illuminated by only two independent sources intersect uniquely. With 
this four-source scheme, photonctric stereo uniquely determines the 
gradient corresponding to an arbitrary inagc point. 

Figure 3-13 illustrates this four-source schene applied to the sample 
sphere of Chapter Z.B. Here, all gradient points corresponding to sampled 
inage paints have been pinned down exactly. Figure 3-14 superimposes on 
figure 3-13 the light source positions and corresponding shadow-lines. 
Those shadow- lines divide gradient space into nine regions, each 
illuminated by a different combination of the four sources- 
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I ? (x,y) = 0.G00 




Ii(x*r> = 0.971 



0-375 



Figure "ii Determining image points whose surface normal Is given by the 
gradient (p t q). Throe superimposed image intensity contours are 
intersected where each contour corresponds to the value at (p.q) obtained 
from three separate reflectance maps. Each reflectance nap characterizes 
the sane lnaglng geometry but corresponds to a different light source 
position. 
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l z (x.>0 = 0.79G 



(x.y) = 0.79G 




I 3 (x,>-) = 0.79S 



- -soa 



Figure 3-12 Determining linage points w9iose surface nor«al directly J««s tho 
viewer- Three superinposcd ioage intensity contours are intersected whore 
each contour corresponds to the value at (0.0) obtained from three separate 
reflectance maps. Each reflectance map characterizes the sane imaging 
geonctry but corresponds to a different light source position. 
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Figure 3-13 Applying photometric stereo to four synthesized images of a 
sphere- The surface reflectance is assumed to be lambertian. The points 

nark the gradients determined at each sampled Image point, (Compare this 
figure with the single image result of Figure 2-10.) 
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indicate which sources can not illinninate that region. 
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Of course, photometric stereo works well on synthesized examples. 
What difficulties night one expect in practice? Nonuniformities in lodging 
hardware, surface photometry and incident illumination, duo. for example. 
to mutual illumination, will all lead to inaccuracies. One can gain some 
insight into how such inaccuracies will affect photometric stereo. 
Figure 3-15 plots the reflectance map contours associated with a two source 
larabertian configuration (spaced 0.1 units apart) when the light sources 
are separated by 9D° . Each region of Figure 3-15 corresponds to a region 
or equal measurement error. The configuration depicted in figure 3-15 is 
quite tolerant of errors for image points whoso corresponding gradient lies 
in the third quadrant of gradient space* On the other hand* slight errors 
in the measurement of intensity for inage points whose gradient lies in the 
Tlrst or second quadrant can lead to HlbltUtlll errors in the solution 
determined for that gradient. Figure 3-16 repeats figure 3-15 but for the 
case when the light sources are separated by 180° . Here, reasonable 
accuracy is achieved in the second and fourth quadrants. 

This analysis is purposely quite qualitative. In any numerical 
solution to an overdetermlned problem, which is subject to measurement 
error, one cannot expect to achieve an exact solution* Figure 3-15 and 
figure 3-16 give son© indication about about how one might select which 
imago combinations to believe and which to hold suspect* In practice, the 
properties of the particular reflectance map must also be taken Into 
account, 
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Figure 3-15 Error regions for two superimposed reflectance naps 
corresponding to lanbertian surfaces illuninaled by separate sources 
rotated about the viewing direction 90° froa each other (g = 37,3 ). Each 
region indicates how an error in intensity ncasurement determines a 
corresponding error in the estination of surface orientation. 
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Figure 3-16 Error regions for two superimposed reflectance naps 
corresponding to lanbertian surfaces illuminated by separate sources 
rotated about the viewing direction 160° fron each other (g » 37.3°). Each 
region Indicates how on error in intensity Deaiurenent detemlnes a 
corresponding error in the estimation of surface orientation. 
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4. TWO CASTING APPLICATIONS 
In order to understand the Inspection requirements of the casting industry, 
it is vital to obtain actual exposure to realistic foundry environments. 
In the course of this work, two foundries were visited that represent two 
extremes of the metal castings industry. The first was an experimental 
foundry for the development of precision, investment-cast turbine blades 
and vanes for aircraft jet engines- The second was a large, batch-oriented 
green sand mold production foundry. 

In this chapter, an attempt is Bade to outline inspection requirements 
representative of the netal casting industry. This serves both as a 
general introduction to casting inspection and as a notivation for the 
particular inspection tasks addressed in subsequent chaptors- 

4.1 PRECISION INVESTMENT CASTING 

Investment casting is the most sophisticated casting procoss used 
commercially. It permits the reproduction of finer detail, greater 
dimensional accuracy, and smoother finished surfaces than can be obtained 
by any other costing process. It permits mass production of complex shapes 
that are difficult or impossible to produce by conventional casting 
processes or by machining. Castings can be produced that require little or 
no finishing for completion, thus minimizing the importance of selecting 
casy-to-machine metals. 

Investment casting allows close control of grain size, grain 
orientation and other solidification conditions which results in close 
control of mechanical properties. The process is adaptable to almost any 
motrtl that can be melted and poured* It is also adaptable to the melting 
and casting of alloys that must be poured in a vacuum or under the 



R. J. Uoodham 112 Two Casting Applications 

protection of an inert atmosphere. 

4X1 ONE PART IN THE MAKING 

The first class of parts to be discussed is that of turbine blades and 
vanes for aircraft jot engines. Early turbine airfoils were forged- Now. 
however, forging has been replaced by investment casting. The reasons for 
this switch are twofold: 

■ A gas turbine becooes more efficient as its operating 
temperature is raised. Alloy development for superior 
high-tenperature performance made forging impractical. 

■ Sophisticated air cooling schenes have been developed 
to permit operation at even higher temperatures. The 
internal passageways necessary for cooling can be 
produced only by casting. 

The particular turbine blades and vanes studied here are made of a 
special high temperature nickel -base alloy using a precision shell 
investment casting process. The initial tooling cost for this type or 
casting is substantial. Consequently, in this environment, one typically 
deals with relatively few part types produced at a relatively high volume 
or, as is the case here, with a part type for which there is no alternative 
manufacturing process available. 

Casting is basically the process of pouring Dolten metal into a mold, 
letting the netal solidify and then separating the part from the mold. In 
certain casting processes, such as die casting, the mold can be used again* 
In other casting processes, such as green sand mold casting, the mold is 
destroyed during part removal. Part reproducibility is maintained in the 
pattern used to produce the nold. 
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In investment casting, both the pattern and the nold arc destroyed for 
each part made. In this case, part reproducibility is naintalned in the 
accurately machined metal dies used Co produce each pattern. Investment 
casting employs a ceraaic mold. The mold is produced by surrounding an 
expendable pattern, nade of wax or plastic, with a refractory slurry. 
After the mold has set. the pottern i* neltud or burned out. creating the 
mold cavity. Finally, the mold is kiln hardened. 

In this particular application, the inside of each turbine airfoil is 
hollowed out in an elaborate pattern- This pattern produces the internal 
passageways used for air cooling. To cast these passageways , a core 
(i.e., a negative inage of the volume to be hollowed out) is itself 
produced out of a silica material. This core is fixed in the wax pattern 
by inserting it between the die halves prior to the injection of tht 
pattern wax. Thus, after the wax has been burned away from the ceramic 
mold, the core becomes part of the mold. 

Let us consider wore detail. The ceramic shell nold is prepared by 
alternately dipping the pattern assembly (pattern and core) into slurries 
of ceramic powders suspended in a liquid, draining the excess, stuccoing 
the wetted surface with a dry refractory grain and drying the resultant 
coating. This process is repeated until the desired nold thickness is 
achieved, generally 1/4" to 1/2*. The initial coating enpioys a slurry 
that is made up of particles finely ground to provide the desired surface 
smoothness. Subsequent coatings contain increasingly coarser refractory 
grains. 

The mold is dried after each dipping operation to allow bonding of its 
individual layers. Moisture removal is regulated by control of wet and dry 
bulb temperatures, air flow and time. The coopleted nold is allowed to dry 
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for a miniDUQ of 24 hours. When the mold has dried out. the wax is burned 
out leaving the silica core suspended in the mold cavity. Final curing of 
tho mold is achieved by kiln hardening. 

Host high-temperature superalloys are melted using a vacuum-induction 
furnace. Vacuum melting reduces the amount of dissolved gases in the 
metal. It also prevents contamination of the metal by the gaseous elements 
present in normal air atmospheres and actually helps purify the metal 
through volatilization of existing impurities. Vacuum Belting reduces tho 
number of resulting inclusions in the casting and thus improves the 
mechanical properties of the metal. 

The molds are readied in a preheat furnace and then moved to the mold 
locker chamber. After this chamber has been evacuated, the interlock 
between it and the main furnace chamber is opened and the mold is moved 
into pouring position. When the specified pouring temperature has been 
reached, the metal is poured into the mold at a controlled rate. The mold 
is then withdrawn back into the mold locker chamber and the interlock 
closed. The mold chamber is then opened to allow removal of tho rilled 
mold* 

After solidification, the ceramic mold material is broken away and the 
casting is cleaned by sand blasting. A Tirst visual inspection is 
performed to catch obvious defects, such as those caused by a crack in the 
mold. After this inspection, the casting is immersed in a strong caustic 
solution to eat away the silica core material still embedded In the 
casting. 

Each casting then undergoes exhaustive inspection. In an application 
as critical as tho production or components for aircraft Jet engines, part 
integrity is of paramount importance. In the United States, castings for 
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aircraft engines are 100% inspected (i.e.. each casting undergoes each 
inspection operation). The inspection operations performed are summarized 
below: 

■ A coordinate gauging machine is used to verify the 
outer dimensions of the casting. 

■ Ultrasonic transducers are applied to selected points 
on the casting to verify the inner dimensions of the 
casting. In this case* it is to verify that the core 
was positioned correctly. 

■ The casting is X-rayed from oultiple views and the 
resulting images are checked for indications of 
inclusions and other internal defects. 

■ Flourescent penetrant inspection is perforned on all 
visible surfaces of the casting to check for surface 
cracks, tears, porosity and inclusions. 

■ The casting is acid-etching to verify grain structure. 

■ Pressure/leak testing is perforned on the internal 
passageways of the casting to verify that none are 
blocked due to residual core nateriaJ or due to excess 
metal resulting from collapsed core aaterlal - 

Detailed quality standards are used for each of the above inspection 
operations. The basic format of these quality standards is to divide the 
casting into areas, each of which is judged by a portion of the standard 
written specifically for it. Important considerations include the absolute 
size of each imperfection, the total number of imperfections and the 
relative spacing between Imperfections. This division of the casting into 
areas is based upon the actual service requirements of the various 
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subsections of the casting. Critical areas subject to high stress. Tor 
example, are inspected to tighter criteria than are areas subject to lesser 
degrees of stress. 

It is important to realize that these quality standards cannot be 
iopleoented by slnple detection devices. Detection devices help delineate 
imperfections. But. the mere presence of imperfections does not lead to a 
PASS/FAIL decision. Rather, imperfections roust be interpreted In the 
context of the part as a whole. The ability to automate existing industry 
standards ultimately requires the ability to relate the imperfections 
detected to a higher-level representation of the object shape. 

4.2 GREEN SAND MOLD CASTING 

Green sand molding is the most versatile and inexpensive casting 
process commercially available. Typical green sand sold foundries are 
batch-oriented, producing many different castings at low to medium 
production volumes* At the particular foundry visited, approximately 40% 
of the castings produced were to meet the internal oanuTacturing demands or 
the parent company. The remaining 60% were cast on a contract basis for 
outside customers. The foundry maintains a library in excess or 100,000 
patterns, any one of which can he scheduled for production. Orders range 
from fewer than 10 parts to as many as 250.000. 

In preparing a casting for production, extensive inspection and 
destructive test facilities are available to eliminate systematic design 
flaws. In each application, however, there is an inevitable trade-off 
reached between design cost and production cost. A decision must be made 
as to whether the production quantity involved warrants further effort to 
modify the casting design. In a batch-oriented foundry, many of the 
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process variables, such as alloy composition, pouring temperature and sand 
composition, qusl be considered fixed since they cannot bo modified from 
part to part. Tims, the most economical compromise often tolerates a high 
rejection rate during production. The rejection rates for castings 
produced at the foundry visited typically vary between 6% and 30%, 
depending principally on the size, shape and complexity of the part* 

Green sand mold casting has the advantage that all the materials 
required can be recycled. The pattern is reusable. The sand used to make 
the mold is reusable. Should the casting be defective, the metal is also 
reusable. Thus, a high rate of rejection is not that expensive. The cost 
of manufacturing a defective part is the cost of the energy required to 
melt the metal and the cost in tine and labor associated with the loss of 
production capacity. 

The principal goal of inspection is to find defective castings before 
investing expensive machining operations on a bad part. Approxlnately 8SX 
to 90% of the casting defects are sufficiently gross In nature that they 
are easily found by unskilled inspectors inoediately following mold removal 
and cleaning* These human inspectors perform a cursory examination of each 
casting as it comes out of the sandblast machines. Ones that pass this 
first visual inspection are sorted into appropriate bins. The remaining 
ones are cataloged by an inspection foreman and then recycled to the 
me] ting furnace. 

A second level of visual inspection is achieved by making it 
profitable for ln-house machinists to find defective castings. Nachinists 
are employed to smooth away residual traces of gating and other minor 
surface Imperfections fron the* castings. These machinists are paid on a 
piecework basis. They are paid for each piece they grind and for each 
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piece they find to be defective. Thus, a few seconds of visual inspection 
can earn as much as several minutes of grinding. In fact, one of the major 
problems the foundry has 15 that a few unnoticed swings with a hamaer can 
make a machinist as much money as a half an hour of ■regular" work. 

Castings that leave the foundry have a reasonable chance of being free 
of defects on all visible surfaces. No Inspection, however, is provided to 
verify the internal integrity of the part. Such castings are not produced 
on a "guaranteed" basis, DcTects are often found during subsequent 
machining operations. A potential disagreement between a foundry and its 
customers centers around the cost of machining defective parts. The 
foundry docs not pay for machining operations performed after a part has 
left the foundry- The customer, however, will often argue that the foundry 
should absorb these machining costs since, after all, the part was 
defective. 

Before presenting the second class of parts discussed, it will be 
useful to give a short introduction to the green sand mold casting process. 
Sand, combined with a suitable binder, is packed rigidly about a pattern, 
so that when the pattern is removed, a cavity corresponding to the shape of 
the pattern remains. Plolten metal poured into this cavity and solidified 
develops a cast replica of the pattern. The sand that forms the mold 
cavity can be readily broken away for subsequent removal of the casting. 

The materials used for pattern making differ greatly in their 
characteristics and therefore in the applications to which they are best 
suited. Patterns can be made of wood, metal or other suitable materials, 
such as wax. polystyrene or epoxy resin. The decision as to what material 
to use depends on the stage or development of the design of the casting, 
the expectod production quantity, the dimension*: accuracy required, the 
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size and shape of casting and the molding process to be employed. 

Green sand molding is the most widely used of all sand molding 
processes* A green sand oold is made of sand, clay, water and other 
materials, "Green" means that the sand mixture remains ooist. Green sand 
©olds are not oven dried or otherwise hardened. The mold is used directly 
without further conditioning. 

The probloos associated with green sand mold casting fall into roughly 
two categories: those associated with metal-to-mold interactions and those 
associated with pouring and feeding the molten netal. Defects during 
production runs are mnimizcd by careful control of sand properties, metal 
handling and pouring rates. Table 4*1, taken froo [ASH 70], illustrates 
typical casting defects that result when sand properties are above or below 
specified limits. The foundry visited maintains an extensive laboratory 
for the testing and control of sand properties. 
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Casting Defects That Result When Sand Properties 

Are Above or Below Specified Units 

Sand Property ~ Casting Defects 

Sand Property Above Liait 
Holsture content Blows, scabs, cuts, rough finish, hot tears, 

porosity, rattails, dirt, high hot strength 

(difficult shakeout). oxide inclusions, 

dimensional inaccuracy 
Permeability poor finish, pinholes, veining, sticky sand, 

raisruns in thin sections 

Green strength Rough finish, difficult shakcout 

Green deformation Scabs 

Mold hardness Blows, scabs, hot tears, difficult shakeout 

Dry strength Hot tears, pinholes, difficult shakeout 

Hot strength Hot cracks, difficult shakeout 

Hot deformation ....,.,, Dimensional inaccuracy 
Conbustiblos content ... Blows, pinholes 

Sand Property Below Linit 
Moisture content Drops, cuts, poor finish, dirt, broken mold 



Permeability Blows, penetration of netal into mold, 

shakeout, scabs 
Green strength Drops, scabs, cuts and washes, pinholes, 

dirt, veining, stickiness, dimensional 

inaccuracy, shrinks 

Green deformation Drops, cuts, dirt 

Mold hardness Drops, cuts, rough finish, penetration of 

netal into nold 
Dry strength Cuts, dirt 

Hot strength Cuts, poor finish, penetration of metal Into 

mold 

Hot deformation Scabs, rattails, dirt, veining 

Conbustiblos content . .. Poor finish, veining, inaccuracy, burn-on, 

cuts, rattalls 



TABLE 4-1 

Metal handling and pouring, on the other hand. Is aore difficult to 
control. All operations are manual. Another recurring problcto in the 
foundry is that metal handlers, also paid on a piecework basis, will try to 
squeeze more mold fills out of each bucket of molten metal than production 
standards call for* 
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Green sand mold casting is not used to produce parts whose aechanlcdl 
properties arc of prime concern- The internal integrity of such a casting 
is difficult to control. On the other hand, preen sand mold casting is an 
economical way to produce a part, out of metal, with desired geooetric 
properties. Defects are principally surface properties which effect those 
oeoiaetric properties, cither functionally or aesthetically, Internal 
defects discovered by later machining are a mild embarrassment to the 
foundry but must he tolerated because there is no economical way to check 
for them. 

Host of the dofects listed in Table 4-1 oanifest themselves as surface 
properties that could not have been the result of a legitimate casting 
operation. Automatic inspection in a batch-oriented foundry would require 
a systen that could detect these surface properties independent of the 
particular part being viewed. 

4.2.1 A SECOND PART IN THE MAKING 

The particular part to be discussed is a snail shuttle eye 
(i.e. f thread guide) used in textile looms. It is one of about 50 
different shuttle eyes produced. Sone differ slightly in part geometry. 
Others differ slightly in part dimensions. It is made of gray Iron and 
measures approximately 5 en by 2.5 cm by 2.5 cq. The size Is worth 
mentioning because this particular casting has a property shared by many 
green sand mold castings. That ls ( if one were only interested in making 
this particular casting, one would not choose a green sand mold process. 
It is a small part of relatively cooplex shape. These two factors cooblne 
to make green sand molding particularly difficult. It is difficult to 
achieve mold stability for a part with such snail, thin sections. It Is 



R. J, Woodham 12Z Two Casting Applications 

difficult to pour the »olten metal so that each mold section Is filled 
uniformly. Rejection rates for the shuttle eye arc in excess of 30X. If 
there were only Interest in making this particular part, it would probably 
be die cast. However, when there is already a green sand mold facility 
available, a high rejection rate can be tolerated. 

Of those rejected, approximately 50% are due to a cold shut defect at 
the boundary between the two najor sections of the casting, A cold shut 
defect occurs when two streams of molten metal neet in the mold cavity and 
tho temperature differential between then prevents the two streams from 
welding together properly. It appears as a lapping or layering on the 
surface of the casting. Figure 4*1 shows a shuttle eye with a cold shut 
defect. 

Approxinately 25X of the rejects are due to pinhole defects. Pinholes 
are small cavities on the surface of the casting caused by bubbles of 
entrapped gas which roach tho surface during solidification. Pinholes can 
occur anywhere on the surface or the casting although they are 
predominantly found on upward surfaces of the larger sections of the 
casting, figure 4~2 shows a shuttle e-ye with a pinhole defect. 

Cold shuts and pinholes are the two most serious defects associated 
with this particular family of parts. The cold shut of figure 4*-! Implies 
a potential mechanical weakness between the two sections of the casting. 
But the major reason for rejection due to either a cold shut or pinholo 
defect is aesthetic- The surface of the casting seen in figure 4-1 and 
figure 4-2 is to be used "as cast." No further machining is called for. 
Thus, those surfaces must "lock right" whether or not any imperfection 
would actually effect the functional performance of the part. 
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Figure 4*1 Image of a shuttle eye with a cold 
of the two major sections of the casting. 



shyt defect at the interface 
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Figure 4-2 Image of a shuttle eye with a pinhole defect. 
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ho other single kind of defect accounts for more than 10* of the 
rejection rate. The two most predooinant remaining defects are raisruns 
(i.e.. incomplete filling of sections of the mold cavity) and dirt blowouts 
(i.e., loose sand in the oold cavity, which, because of its wetness, 
explodes on contact with the oolten petal). Nisruns are nore difficult to 
dfltect because, in order to decide that a particular section of the mold 
cavity has not filled completely, it is necessary to have some knowledge or 
what the part ought to have looked like. Dirt blowouts, on the other hand, 
arc always quite obvious since the "explosion' of wet sand tears up a large 
portion of the part- A more complete catalog of typical defects in oetal 
castings is presented in Appendix B. 
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iS AN ASIDE: THF FOUNDRY AS WORK ENVIRONMENT 

Unless one has worked in a foundry, it is inpossiblo truly to 

appreciate wliat a hostile environment it is. There are the obvious 

physical demands and dangers* There are also more subtle hazards. The 

following quote is taken from a chapter on sand molding in [ASH 70]: 

"Silica dust from foundry operations can produce 
silicosis if tAcre is sufficient exposure, in terms of time 
and concentration . to free crystalline it lie* dust of 
particle size below five microns* Two to twenty years 
(average 10 years) ore required to produce a case of 
silicosis when dust concentrations greatly exceed the 
maximal a I loaable . . , 

...The oil no-bake binders for sand molds consist of 
three parts: a modified linseed oil. a metallic drier, and 
an aromatic tsocjforttfte. Soraettnos the last two creote a 
health hazard. 

Cobalt naphthenate (6% CoJ, tAe drier most commonly 
used In this binder, is slightly toxic. Cobalt compounds 
can produce contact demott tis and sensitization. Lead 
naphthenate is also used as a drier. Lead can be absorbed 
by the skin and cause lead poisoning, especial lit when 
mineral oil, which may induce dermatitis and irritation, is 
used. 

Under ceriain conditions of poor ventilation, the 
amount of aromatic isocyanate (methylene di phenyl 
diisocyanate. 1901) vapor liberated in shakeout operations 
may be a significant health hazard. Operators of shakeout 
deuices and of overhead cranes in tfte area should wear an 
approved organic vapor respirator. Persons with known 
asthmatic history should not be employed on operations that 
release high levels of HOI vopor. To reduce decomposition 
of MU t neither the sand nor the binder should be heated 
above 125 F* Staining of the hands by HOI can be reduced 
if molders use protective creams resistant to oils and 
solvents and wear rubber or plastic-coated canvas gloves. 9 * 
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5. GRAIN SIZE ESTIMATION 
Grain structure is an important process variable in «any casting 
applications. A gas turbine, for example, becomes more efficient as Its 
operating temperature is raised- Methods are continually being sought to 
enhance the hlnh-teaperature pcrfomance of turbine blades and vanes. 
Grain boundaries play a critical role in determining the high-temperature 
mechanical properties of a netal. Increases in strength generally are 
achieved with a reduction in ductility, and, in high-temperature alloys, 
one nanifestation of this phenomenon is a reduction In creep strain prior 
to failure. Examination of failed material shows that croep cracks are 
Initiated along grain boundaries that make a large angle to the principal 
stress direction [Sullivan 1976]. Consequently, both grain size and 
preferred grain orientation significantly alter the mechanical response of 
a metal, Superior performance is achieved by carefully controlling grain 
structure in solidification processing. 

The most important factor determining grain structure is the heat flow 
pattern during solidification. The principal determiner of grain size is 
the rate of cooling during solidification. The faster the cooling the 
finer will be the resultant grain structure. Areas of a casting that cool 
very rapidly, and thus have a fine grain structure, are terned chitt zone*. 
The principal determiner of grain orientation is the direction of heat 
transfer during solidification. Grains tend to grow parallel to the 
direction of heat flow. Those that have an elongated aspect ratio are 
called columnar oraitis. 

Castings which have no preferred grain orientation are called equiaxed 
castings. Castings which try for superior mechanical performance by giving 
a preferred grain orientation with respect to the principal stress 
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direction are called dtrcctionatty sotidijied (OS) castings. The turbine 
blades and vanes considered in this work are examples of eguiaxed castings. 
In the manufacture of these components, three qualitative grain properties 
are considered essential to achieve the required resistance to crack 
propagation: 

1. Grain nust be sufficiently large. 

2. Grain must not have a preferred orientation. 

3. There must be no sharp demarcation lines between regions 
of different grain size. 

Considerable effort Is required to produce turbine blades and vanes 
with grain structure satisfying these three properties* Heat flow during 
solidification naturally occurs out Trom the leading and trailing edges of 
the airfoil. The columnar grain which forms as a natural consequence of 
this boat flow is particularly undesirable since the Mjor stress the 
airfoil will encounter in service occurs precisely along this direction. 
Turbine blades and vanes are 100X inspected to verify grain structure. 

In other casting applications, a different grain structure may be 
desired. In aluninum castings, for example, fine grain is generally 
desirable. The coarseness of porosity is proportional to grain size. 
Consequently, porosity is finer and less hornful in fine grain aluainua 
castings. Also, certain mechanical properties, such as tensile strength, 
are usually superior for fine grain aluainun castings- Finally, for 
aluminum alloys, fine grain ninioizes shrinkage, causing castings to bo 
sounder. 
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5.1 ESTIMATING THE AVERAGE GRAIN SIZE OF METALS 

The problem in nondestructive grain size estimation Is to determine 
the average grain size of a metal structure, a three-dimensional property, 
from grain cross sections observed on a two-dimensional surface passing 
through that structure. It is important to recognize that such an 
estimation of grain site can never be an exact measurement* A octal 
structure is an aggregate of thr*e-dim«nsional crystals of varying sizes 
and shapes. Even if all those crystals were identical, the grain cross 
sections, produced by a random surface of observation through such a 
structure, would have a distribution of areas varying froa a maximum value 
to zero, depending upon where the surface cuts each individual crystal . 
Clearly, no two fields of observation can be exactly the same. In an 
equiaxed casting, the size and location of grains is normally completely 
random* 

The American Society for Testing and Materials recommends three basic 
methods for grain size estimation as part of A5TH Designation: E 112-74 
■Standard Methods for Estimating the Average Grain Size of Hotels" 1 : 
1- Comparison Procedures: 

Comparison procedures aro used for cotapletely 
recrystalllzed or cast materials with equiaxed grain. 
They involve a direct comparison of a representative 
field of the test specimen with an appropriate 
collection of standard grain size pictures. The 
standard which most closely matches the test specimen is 
selected and the corresponding grain size is recorded. 
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Comparison procedures are considered convenient for 
human inspectors and sufficiently accurate for most 
comaercial purposes. However, experience has shown that 
unless the appearance of the standard grain size series 
reasonably well approaches that of the sample, errors 
may occur. Thus, a particular standard grain size 
series does not adapt well to different alloys or 
different methods of specimen preparation. 
2. Planimetric Procedures: 

Planimetric procedures involve counting the number 
of grains in a known area of the test specimen, usually 
a circle or rectangle. The sum of all the grains 
included completely within the known area plus one half 
the nunber of grains intersected by the circumference of 
the area gives the number of equivalent whole grains, 
measured at the magnification used, within the area. A 
simple computation. based on an assumed formal 
relationship between grains per unit area and grains per 
unit volume, is used to convert this count into en 
estimate of the number of grains per square millimeter. 
Planimetric procedures are mor« accurate than comparison 
procedures. Accuracy falls rapidly, however, when grain 
deviates from an cquiaxed structure. 
3* Intercept Procedures: 

Intercept procedures involve counting the number of 
grains crossed by a standard pattern applied randomly to 
the test specimen, usually one or more straight lines or 
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circles. The length of th* test pattern divided by 
count of the nunber or points where the test pattern is 
cut by a grain boundary gives the mean intercept 
distance, also called acan free path or Heyn intercept. 
It can be shown that this distance is an unbiased 
estimate of the mean intercept distance within the solid 
material in the direction , or over the range of 
directions. measured- 
Intercept procedures are reconmended particularly for structures 
consisting or elongated grains or for structures containing a nixture of 
actual grain sizes. In the absence of a specific engineering judgoent to 
the contrary, the intercept size is always considered to be the defining 
grain size value. Indeed, ASTK Connitlce E-4 on Metallography, under whose 
Jurisdiction the consensus standards for grain size estimation are written, 
adopted the following as their official position on ■easurement of grain 
size: 

TAe rejeree method should be the Intercept weifrod, and the 
defining equation for groin size number should be that 
presented in Methods t 112** 

" 1074 Report of Committee E-4 on Metallography 
ASTH Proceedings. Volume 75, 1975 

Intercept procedures using circular test patterns automatically 

compensate for departures from equiaxed grain, without giving too much 

weight to any local portion of the field. Ambiguous intersections at the 

ends of linear test lines arc elininated. Circular intercept procedures 

arc considered most suitable for grain size estimation in quality control* 
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In light of these recommendations, the method explored for estimating 
the average grain si2e of equiaxed turbine blades and vanes was the three 
circle (Abrams) procedure specified in section 1K4 of ASTfl Standard 
E 112-74, The goal of this work , is to demonstrate that a machine vision 
system is capable of the performance and accuracy called Tor by the 
existing industry standard. First, the Abrams procedure is outlined and 
then the details or the algorithm developed to implement it are presented. 
Finally, grain size estimation froa arbitrary views of curved surfaces Is 
considered. The flexibility required to perform this inspection is 
achieved by relating grain size measurements in an image to the 
corresponding viower-centered representation of surface shape. 

5.2 THE THREE CIRCLE (ABRAMS) PROCEDURE 

The three circle (Abrans) procedure is en intercept procedure in which 
it is assumed that the surface of observation is planar. This assumption 
of planarity is implicit in the conputation used to infer the 
throe-dimensional grain size of an eQUiaxed casting from measurements on a 
two-dimensional surface of observation. The test pattern consists of three 
concentric, equally spaced, circles designed to cover a total circuaference 
of 500 milliaeters on the sample surface- A little matheaatlcal figuring 
will show that the diameters of these circles are 26,53 m, 53*05 ma and 
79,58 no respectively. The procedure is as follows: 

1. Perforn a cursory exanlnation of the specimen to roughly 
estinate its grain size. Using this rough estimate, 
select an image magnification that will yield 
approxlnately 100 gram crossings over the 500 era three 
circle test pattern. 
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2. Randomly select one field for measurement and apply the 
test pattern to that field- If the count of grain 
crossings for the first application is less than 70 or 
nore than 140 t discard the result and adjust the imago 
magnification accordingly. Repeat this step until the 
count is In the acceptable working range* 

3. Randoaly select four more fields for measurement and 
apply the test pattern to each field. A total of five 
such counts is considered sufficient to compute grain 
size to within 1/2 a standard ASTil grain number. If 
additional accuracy is required, additional fields may 
be sampled. 

The grain size, as neasured above, is indicated as a count of grain 
crossings over a known path length - Such a value is inconvenient for 
subsequent use. Hence, this count is normally reexpressed In terms of 
industry standard quantities such as nominal diameter. Ferct's diameter. 
Intercept size, specific surface, grains per unit volunc, or ASTF1 
micro-size or macro-size number. 

The ASTM grain number is defined as follows; 

C = 10.0 - Z log2(L) 
where L is the mean intercept distance which, for the thro© circle (Abrams) 
procedure, is given by: 

* a length of path (in millimeters) 
total number of grain crossings 

If L is ilctoroinod in millimeters at an inage magnification of 100X. then 

the resulting C is called the tvricro-grcin number* If L is deterained in 

millimeters at an inage magnification of IX, then the resulting C is 
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called the mocro-grctn mmbcr. The program discussed below which 
implements the Abrams procedure has sufficient accuracy to determine the 
statistically correct A3TH grain number for saiiples in the recommended 
working range. The program was tested and calibrated using sample plates 
available from the American Society for Testing and ttaterlals as an adjunct 
to standard E 112-74- When a particular sample spiclnen was made to fall 
outside the recommended working range, the program was generally robust 
enough to be able to suggest an appropriate change to image magnification 
to bring the sample within the desired range, 

5.3 A PROGRAM FOR DETERMINING ASTM GRAIN NUMBER 

The initial discussion presented here focuses on the problem of 
applying the three circle (Abrams) procedure to images of a planar surface 
viewed in a direction normal to that surface. The reason for this Is 
twofold. First, the applicable grain size measure, as defined in ASTM 
standard E 112-74, demands that the viewed surface be planar. Second, by 
viewing in a direction nonwl to the surface, one avoids the additional 
mathematics rcquirod to account for the foreshortening of path length due 
to an oblique viewing angle. 

The program discussed makes use of simple edge mask filters to 
determine grain boundaries crossed by a circular test pattern. Unlike 
other simple detection schemes, however, the program does not make 
decisions based upon any single measurement. Bather, grain crossings are 
detcroined by comparing the relative response over an ensemble of different 
mask sizes- This makes the program usable over a broader range of image 
magnifications and grain sizes and, at the same time, makes it less 
susceptible to false indications due to noise. 
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First, tho method used to determine grain crossings along a circular 
path on the surface Is summarized. Second, the performance of the each 
portion of the program is analyzed in more detail using synthesized data. 
Finally* an example of the program applied to an etched section of an 
actual turbine vane is provided. 

The algorithm for determining grain crossings along a circular path on 
the surface can be summarized as follows: 

1. DATA ACQUISITION: 

The lodge is sampled along a circle centered at image point (Xc ,y c ) and 
corresponding to a surface dianetcr of d millimeters. Tho resulting 
intensity values arc stored as a one-dimensional (circular) array called 
INTENSITY. 

2. INITIAL DATA PROCESSING: 

Three different edyo detection filters are applied to the intensity 
values. Each edge filter is of the Font: 

Fn(i) = [I(i*n + I(i*2K..*I(i+m)] - [I(i-l}*I(l-2)+. - .♦!( i-m)] 
where I{i) denotes the i th element of the array INTENSITY- The results 
are stored as three net* one-dimensional (circular) arrays, called 
FILTER1, FILTER2 and FILTERJ respectively. Each FILTER array 
corresponds to a different choice of the value m. 

3. DATA COMPRESSION: 

Each array FILTERn Is processed to produce a list U of tho form: 

Ln ■ ((TYPEl INDEX1 VALUE1) ... (TYPEk INDEX* VALUE*)) 
Each clement of the list l- denotes a local extremum in tho 
corresponding FILTERn array, TYPE is MAXMUfl or HIMIHUH, INDEX is tho 
index of the extremum in array FILTERn and VALUE is the height of the 
extremum* The three lists Li f Lz and L3 are then merged Into a single 
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list 

L B ((TYPE1 1NDEX1) (TYPEZ INDEXZ) ... (TYPEn INDEX*)) 
Again, TYPE is MAXlflUH or HINIHUH and INDEX is the Index of the 
extremum. L contains an element for each local extreoum found In either 
Li i Lz or L3 . 
4, SYNTACTIC ANALYSIS: 

The list L, together with the lists Lj f L2 and L3 are passed off to a 
syntactic parser. Basically, the job or the parser is to look at each 
extremum In L and decide whether it corresponds to an actual grain 
crossing. The parser consists of a collection or parsing routines which 
may accept a given extrenum as a grain crossing, Each parsing routine 
is given initial matched pointers into Li, L* and La. It can then look 
both forward and backward fron those pointers using any or all of the 
lists Li ( La or L3 to conpare the responses measured by the FILTER1, 
FILTER2 and FILTER3 arrays. If the responses are consistent with the 
type of edge being parsed, then the peak Is accepted as a grain 
crossing. 

Data acquisition is straightforward when the surface of observation is 
planar and Is viewed in a direction normal to the surface. Circles on the 
object surface will project to circles in the iaage. To compute the A5TR 
grain number, it is only necessary to calibrate the imaging hardware with 
respect to magnification* Unfortunately, the slow-scan vidicon camera 
{Spatial Data System 106) used to digitize inages for this work has a 
different spatial resolution in the. inage x-dircction than in the image 
y-direction. Thus, two calibration measurements are perforoed. One 
determines A, the number or picture eleaents per millimeter in the 
x-direction and the other determines B. the number of picture elements per 
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millimeter in the y-directlon. For a particular imaging situation, this 
calibration can either be done manually or automatically using ah 
appropriately scaled "ruler* in the field of view. Thus, In order to scan 
a circle on the surface centered at image point (Xc»y<) and of diameter d 
mllllmcterSt one actually scans the i range ellipse described parametrically 
by: 

x(fl) = x* ♦ A r cos(tf) 

y(fl) = Yc * B r sin(ff) 
whore r = d/Z, the radius of the circle in Millimeters and < B < 2n. 

The raw intensity values obtained with the vidicon camera are 
typically quite noisy and hence difficult to interpret directly. Simple 
edge mask filters enhance effects due tc edges and suppress effects due to 
noise. [Shirai 75] used the same filter technique in his program for 
tracking edges of polyhedra. [Harr 76] uses a similar type of edge mask, 
extended in the second dimension, as one of the two basic operations in the 
computation of the prinal sketch. 

The contribution of the present work is not so much in the use of the 
simple edge nask filters but in the use of multiple filters of varying 
support widths. Tor any particular choice of *, the filter array computed 
by: 

Mi) * [I(i+l)+l<i+2)+...+I(i+n)] - [i(i-i)*i(i-z)+... + Ki-m)] 
forces an inevitable trade-off between noise immunity and resolution. For 
small n, there will be a greater number or raise peaks due to noise- For 
Largt m, there will be a loss or resolution as closely spaced edges become 
smeared tngnther, especially in situations where adjacent edges have 
intensity gradients with the same sign. 
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For a given image, it is usually possible to hand-pick a choice for n 
that works reasonably well. In the first version of the grain size 
estimation progran, a single FILTER array was computed corresponding to * 
single choice of m. Unfortunately, no single value of n proved satisfactory 
over the range of grain sizes called for in ASTH standard E 112-74, With 
three filter arrays, corresponding to three choices of m, superior noise 
innumity and resolution is achieved. The existence of a peak in one or the 
three filter arrays is used to hypothesize the presence of an edge. There 
way not be ftatchlng peaks in the other two niters arrays, but if the 
values of the other two filter arrays at the position of the peak are as 
predicted by the hypothesized edge, then the existence of that edge can be 
reliably asserted. 

Figuro 5-1 illustrates edge nask preprocessing applied to a single 
ideali20d grain cro«iftg. Figure S-1U) shows the intensity profile 
corresponding to a step in intensity. Figure 5-l(b>, figure 5-l(c) and 
figure 5-l<d» show, respectively, the filter profiles corresponding to 
choices of mj , n? and no where the ratio mi im? :to is 1:2:3. The step in 
intensity produces a simple peak in each filter prorile. The heights and 
widths of the corresponding peaks are also in the ratio mi :m?:iQ . 

Figure 5-2 illustrates edge mask preprocessing applied to the sane 
Idealized grain crossing of figure 5-1. In this case, however, each 
intensity value in figure 5-ZU) has been perturbed by adding a white noise 
of amplitude h, where h is the height of the original step in intensity. 
Figure 5-2<b), figure 5-2(c) and figure 5-2{d) show, respectively, the 
filter profiles corresponding to choices of b ]( m? end n* where the ratio 
m :m?:«3 is again 1:2:3. All three filter profiles have successfully 
isolated the edge due to the step in intensity from the effects due to 
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Figure 5-1 Edge mask preprocessing applied to an idealized intensity step. 
Figure 5-l(a) Is the intensity profile. Figures 5-l(b), (c) and (d) are 
the mask results obtained when the widths of the casks are in the ratio 
1:2:3- 
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noise. The height of each peak in the filler profiles preserves the 
approximate ratio 1:2:3- So Tar. there seuis to be no advantage to having 
computed throe separate filter arrays. 

Figure 5-3 illustrates the superior noise imunity that can be 
obtained using three separato filter arrays. Figure 5-3(a) again 
corresponds to the idealized grain crossing of figure 5-1 perturbed by 
white noise. In figure 5-3{a) t however, the amplitude of the noise was 
increased to 1.6ft. where h is egain the height of the original step in 
intensity. Figure 5-3(b), figure 5-3(c) and figure 5-3(d) show, 

respectively, the filter profiles computed using the same choices for mj t 
ma and m? used in figure 5-2. There is a peak In tho FILTER1 array 
corresponding to the underlying step in intensity. This peak, however, 
hardly "stands out - froa the other peaks due to noise round in FILTER!, 
Mote, however, that the height of the peaks in the three filter profiles 
corresponding to the underlying step in intensity preserve the approximate 
ratio 1 :2:3. Other false peaks in FILTER1 do not preserve this ratio 
across the three filter arrays. Thus, they can be reliably rejected. 

One might argue that the value chosen for mi is simply too small for 
the noise levels present in the example of figure 5-3- The next two figures 
illustrate that such a snail value for mi is necessary to resolve closely 
spaced edges. Figure 5-4 illustrates an idealized example corresponding to 
two closely spaced grain crossings such that the step in intensity across 
each boundary has tho saoe sign- Figure 5-4(a) shows the intensity 
profile. Figure 5-4(b), figure 3-4(c) and figure 5-4(d) show, 

respectively, the filter profiles corresponding to the sane choices of nj , 
nt2 and cd used above. FILTER1 clearly resolves the two steps in intensity. 
FILTER2 has begun to smear the two peaks together. FILTERS has smeared the 



R. J. Voudhiin 



141 



Grain Size Estimation 




Figure 5-2 Edge »ask preprocessing applied to a noisey intensity step. 
Figure 5-2(a) is the intensity profile. Figures 5-2{b), (c) and (d) are 
the nask results obtained when the widths or the masks are In the ratio 
l:Z;3. Each mask has isolated the step and the approxioate ratio of the 
peak heights is preserved. 
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Figure 5-3 Edge mask preprocessing opplled to a noisier Intensity step- 
Figure 5-3(a) is the intensity profile. Figures 5-3(b), (c) end (d) are 
the mask results obtained nhen the widths of the masks are in the ratio 
1:2:3. In Figure 5-3(b), the puak dua to the step is no higher than peaks 
due to noise. Nevertheless, the approxinate ratio of the peak heights at 
that point confirms the presence of the step. 
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two peaks together to the point that it is no longer possible to resolve 
then. Nevertheless, the values in the three filter profiles, at each peak 
position in the FILTER1 profile, preserve the approximate ratio 1:2:3. 
Thus, the presence of two separate grain crossings, as hypothesized by the 
two peaks in the FILTER) profile, can be reliably asserted, not due to the 
presence of corresponding peaks in the FILTER2 and FILTER3 profiles but 
rather flue to the corresponding values In the FILTER2 and FILTER3 profiles. 

Figure 5*5 illustrates the sane example as figure 5-4 with the 
addition of white noise of amplitude h t where h is the height of each stop 
In intensity. Again, the presence of separate grain crossings is 
hypothesized by the two peaks in the FILTER1 profile and verified by the 
corresponding values in the FILTERZ and F1LTER3 profiles* The computation 
of filter profiles corresponding to three different choices of a provides 
tho basis for a method for determining grain crossings which gives superior 
noise imnunity without a corresponding loss in ability to resolve closely 
spaced edges. 

Initially, however* coniputing the three FILTER profiles simply 
generates three new arrays of numbers. The next important stop in the 
iiluurithn is to convert these FILTER arrays into a form more amenable to 
the kind of analysis presented informally in the discussion of figure 5-1 
through figure 5-5 above. 

The computation of each list Ln . where 

U = {(TYPE1 INDEKl VALUED -.- (TYPEk INDEX* VALUEk)) 
serves two functions. First, it represents each FILTER array as a set of 
symbolic assertions which serves as the appropriate input to the syntactic 
parser. Second, it can also be properly viewed as a data compression 
process. The list L carries Information only about tho position, height 
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Figure 5-4 Edge mask preprocessing applied to two closely spaced idealized 
intensity steps (of the sane sign). Figure 5-4(a) Is the intensity 
profile. Figures '-4(b). (c) and (d) are the mask results obtained when 
the widths Of the masks are In the ratio 1:2:3. 
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Figure 5-5 Edge mask preprocessing applied to two 
intensity steps (of the sa»e sign)- Figure 5-5 
profile* Figures 5-5(b). (c) and (d) are the «sk 
the widths of the masks are in the ratio 1:2:3- 
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and type of each local extrcmum in the corresponding FILTERn. In fact, one 
can throw away the INTENSITY and FILTER arrays at this point. It is true 
that subsequent analysis will require values for F n (i) where i is not a 
local extremiiRi of the particular Fa, but these values can be successfully 
approximated as a linear interpolation between the values at the extreoa 
neighboring i. Saying the same thing again, each U used in the subsequent 
syntactic analysis corresponds to a "stylized* F.(i) for which neighboring 
local extrema are joined by straight lines. In the current implementation, 
the FILTER arrays are available so that it is nost convenient to refer back 
to the original arrays as required- A schenc based on linear interpolation 
between peaks was also tried with no change in accuracy. 

The three lists Li ■ L* and L3 are also merged into the single list 
L = ((TYPEi INDEXi) (TYPE? INDEX*) ... (TYPEi INDEXj )> 
The list L collects, in one place, all the peaks found in the arrays 
FILTERI, FILTER2 and FILTER3. This list feeds the parser candidate 
positions for possible grain crossings, The lists Li , L? and L3 are 
available to each individual parsing routine for examining particular 
features within each filter profile and for comparing responses across 
filter profiles. The process of parsing consists of passing successive 
peaks from L off to the individual parsing routines each of which defines a 
class of edge that will be accepted by the parser. When invoked, each 
individual parsing routine is also passed pointers to the closest matching 
peak in each of the lists Ll» 1*2 and La* These individual routines pay look 
forward or backward in any or all of the lists Ll, L2 and La in an effort 
to parse a candidate peak. A peak is rejected as an edge if it is rejected 
by all of the parsing routines present in the systen. A peak is accepted 
as an edge when it is accepted by one of the individual parsing routines 
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present in the system. If a peak is accepted by one of the individual 
pursing routines, it Is the responsibility of that routine to position the 
input pointer correctly past any adjacent peaks that have already been 
accepted as part of the edge. Parsing is conplete when there are no »or* 
candidate peaks to consider. 

The simplest example of a parsing routine is STEP-UP which accepts 
grain crossings corresponding to a simple upward step in intensity like 
those illustrated in figure 5-1 through figure 5*5. STEP-UP is invoked 
whenever a peak or type HAXIMUfl occurs in the list L. STEP-UP. like all the 
individual parsing routines, is passed the four arguments: i, ll, 12 and 
13 where i is the index of the candidate peak or typo HAXIP1UM from L and 
il. lZ and 13 are. respectively, the indices in FILTER!, FILTER2 and 
FILTERS of the peak nearest to i oT type NAXIMffl. STEP-UP accepts a grain 
crossing at index i if: 

(1) Fi{i) > (THRESHOLD * m, ) 

(Z) Fi(i)/mi, F?{l>/B2 and Fad)/^ have 'about the sane 
value". 
STEP-UP is so simple that It does not moke use of any forward or backward 
analysis of the three lists Li , Lz end La ♦ Another parsing routine, 
PULSE-UP, discussed below, does make use of both forward and backward 
analysis, THRESHOLD is a global threshold which establishes the minimum 
step size to be considered significant. Alas, it was not possible to rid 
the program entirely of thresholding operations. The value of THRESHOLD, 
however, Is not critical. The actual value used in the program examples is 
THRESHOLD ■ 3 which is slightly below the average noise level of the 
Imaging hardware, tnoa&ured to be about 5 gray levels. The presence of a 
threshold is required to rule out spurious combinations of peaks, at the 
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love! or noise, which happen to satisfy (Z) above. Finally, a set of 
numbers {v, ) is said to have "about the same value* if the absolute value 
of the dirrorcnce between any two of then is less than one half the 
absolute value of the one with largest magnitude. This definition 
corresponds to a rather ad hoc definition of qualitative equality. A more 
rigorous definition would have to take into account the signal to noise 
properties of the imaging hardware. Again, however, the particular 
definition of "about the same value" is not critical. In general, for 
choices of mi , my and ro in the ratio 1:2:3, there is a good separation 
between peak heights of "about the ratio 1:2:3" and peak heights 
corresponding to noise. The actual boundary chosen was not of critical 

importance. 

The above parsing routine STEP-UP, together with a sinllarly defined 
routine STEP-DOWN, is sufficient to perforn the syntactic analysis required 
to apply the three circle (Abrams) procedure to alloy specimens that can be 
prepared using a contrast etcA technique. In a contrast etch, the cross 
section of each grain is etched so that surface reflectance becomes a 
function of grain orientation. Thus, Individual grains appear as regions 
of homogeneous intensity. Grain boundaries appear as contrast boundaries 
between adjacent regions. Other alloy specimens are nor* appropriately 
prepared using a Jtot etch technique. In a flat edge, the cross section of 
each grain is not affected. Rather, the boundaries themselves are etched. 
Regions within the confines of a grain boundary are unaffected while the 
grain boundaries themselves appear as lines on the specimen. 

The turbine blade and vane specinens used in this work were prepared 
using a contrast etch technique. On the other hand, the standard plates 
obtained froa the American Society for Testing and Materials correspond to 
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samples prepared using a flat etch technique. Thus, additional parsing 
mutinos were also included to handle grain crossings which appear as lines 
in the image, Figure 5-6 illustrates edge mask preprocessing applied to a 
single idealized grain crossing from a specimen prepared using a flat etch 
technique. Figure 5-6[a) shows the intensity profile corresponding to a 
narrow pulse in intensity. Figure 5-6(b)» figure 5-6(c) and figure 5-6<d) 
show, respectively, the filter profiles corresponding to choices of oij , m2 
and ni} where the ratio raj: 02: no is once again 1:2:3, The pulse in intensity 
produces a doublet in each filter profile. The positive peak corresponds 
to the leading edge of the pulse and the negative peak corresponds to the 
trailing edge of the pulse. Note that if the width, w ( of the pulse is 
small compared to m f n? and ma, then the height of each positive and 
negative peak is constant in each or the FILTER1, FILTER2 and FILTERS 
profiles. It is. In fact, equal to the area under the pulse. Note, also, 
that the transition from positive peak to negative peak occurs over the 
width w. again independent or which filter array is considered. 

Figure 5-7 illustrates edge mask preprocessing applied to the same 
idealized grain crossing of figure 5-6. In this case, however, each 
intensity value in figure 5-7(a) has once again been perturbed by adding 
white noise of aoplitude 0,5h ( where h is the height of the pulse in 
intensity* Figure 5-7fb), figure 5-7(c) and figure 5-7(d) show, 

respectively, the filter profiles corresponding to the same choices of mi, 
mz and m* used above. The same qualitative observations hold. The pulse 
In intensity produces a doublet in each filter profile. The magnitude of 
each positive and negative peak is independent of the choice of m\ t 02 and 
mi . The width of the transition from positive peak to negative peak in each 
filter array approximates the width of the original pulse in intensity. 
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Figure 5-6 Edge Bask preprocessing applied to an idealized pulse (two 
closely spaced intensity steps or opposite sign). Figure 5-6<a) is the 
intensity profile. Figures 5-6(b). (c) and <d> are the mask results 
obtained when the widths of the masks are In the ratio 1:2:3- 
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These observations form the basis for the parsing routine PULSE-UP used to 
accept grain crossings from a specimen prepared usinfl a flat etch 
technique. 

PULSE-UP is alse invoked whenever a peak of type HAXIHUH occurs in the 
list L. PULSE-UP is passed the four arguments: 1, 11, 12 and 13 where 1 
is the index of the candidate peak of type HAXIHUfl froa L and 11. 12 and 13 
are. respectively, the indices in FILTER1, FILTER2 and FILTEH3 of the peak 
nearest to i of type MAXIMUM. PULSE-UP accepts a grain crossing at index i 
ir: 

(1) The value. V*. of F1LTER1 at the closest previous peak. 
I*, of type MAXIMUfl is greater than (THRESHOLD * mi). 

(2) The value, V- t of FILTER1 at the closest following 
peak. I-. of type HI Ml BOH is less than 
-(THRESHOLD * mi), 

(3) In going backward in LI from the positive peak at I+, a 
wilue below (THRESHOLD ■ «i ) or V+ - (THRESHOLD * ai ), 
whichover is greater, is encountered before a value 
above V+. 

(4) In going forward in LI from the negative peak at l- t a 
value above -(THRESHOLD ■ mi ) or V- + (THRESHOLD ■ mi). 

whichever is less. Is encountered beTore a value below 
V-. 

(5) Let the distance between the HAXIHUM peak at I* and the 
RIMIflUfl peak at I- be w. Then: 

a) w is less than m* 

b) FiU+)/nin(w,mi}» F?( HJ/ntnfw.nz} and Fa(I+)/w have 
"about the same value". 
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Figure 5-7 Edge nask preprocessing applied to a noisy pulse (two closely 
spaced noisy steps of opposite sign). Figure 5-7{a> is the intensity 
profile. Figures 5-7(b), (c) and (d) are the mask results obtained when 
the widths or the masks are in the ratio 1:2:3. 
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c) Fj(I-)/min[w.irn ), fe{ I-)/min(w,i£ 1 and F3(I-)/w have 
"about the sane value". 
THRESHOLD is the sane global threshold referred to in the definition of 
STEP-UP. (1) and (2) rule out spurious doublets occurring at the level of 
noise. (3) and (4) make sure that the doublet in FILTER! has well defined 
leading and trailing edges. Otherwise, a local *ero crossing which 
actually is a part or a more global feature may incorrectly be accepted as 
a grain crossing. (5a) Bakes sure that the separation between the MAXIMUM 
and MINIMUM peaks is small enough, compared to the values of mi , m? and ma. 
so that the doublet can be considered as corresponding to a single pulse in 
intensity, rather than to two separate steps in intensity. (5b) and (5c) 
are the key steps in PUL5E-UP. (1), (2). (3), (4) and (5a) servo only to 
eliminate cases which otherwise behave as if they corresponded to a pulse 
In intensity. (5b) and (5c) check, respectively, that the hoight of the 
MAXIMUM peaks and the height of the HINIMUM peaks in each of FILTER1. 
F ILTERZ and F ILTER3 have "about the same value* , In the discussion of 
figure 5-6. it was pointed out that if w is snail coapared to mi, m? and 
ma. then the magnitude of each positive and negative peak Is constant In 
each of the FRTEIU, FILTER* and FILTERS arrays. Here, a little oore 
generality is allowed. If w Is less than n , then the expected height of 
each peak in FILTER! is h * w. On the other hand, if w is greater than n , 
then a pulse or width w looks to FILTER1 like two separate opposite steps 
in intensity and the expected height of each peak in FILTER1 is h * ra, . 

The above parsing routine PULSE-UP. together with a similarly defined 
routine PULSE-DOUN. is sufficient to perform the syntactic analysis 
roqni i i to apply the three circle (Abrams) procedure to alloy specimens 
that arc prepared using a flat edge technique. PULSE-UP and PULSE-DOWN are 
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also useful for alloy specimens prepared using a contrast etch- The 
routines PULSE-UP and PULSE-DOWN give the progran superior ability to 
resolve closely spaced intensity steps in situations where the adjacent 
steps have intensity gradients with the opposite sign. In the case of 
samples prepared using a contrast etch, a peak accepted by PULSE-UP or 
PULSE-DOWN corresponds to two separate grain crossings* 

A real example is now presented. Figure 5-8(a) shows a macro-etched 
section or a turbine airfoil. The airfoil has been prepared using a 
contrast etch. Superlnposed upon the grain pattern is the three circle 
test pattern used In the analysis. For illustrative purposes, the test 
pattern is given ^ three concentric circles in the iaage rather than as 
the corrected three ellipses. Crosses marked along the three circles 
indicate positions where grain crossings were detected by the program. 
Figure 5-8{b) illustrates the results obtained when the program analyred 
the middle circle of figure 5-8{a). 

Figure 5-8(b) consists or rive graphs. In each case, the abscissa 
corresponds to angular position (0 to Zn radians) along the test circle. 
The rirst graph shows the intensity values obtained with the vidlcon 
camera. The next three graphs show the filter values computed for case 
mi = 5, m ■ 10 and no = 15 respectively. The fifth graph simply marks, 
for reference purposes ( the points at which grain crossings were detected 
by the parsing routines discussed above. 

A good example of how the use of three different choices of n helps 
can be seen in the second and third grain crossings or figure 5-B(b)» 
counting fron the left. The FILTER1 grajjh does have two negative peaks at 
those points but their oagnitude is not much above that of the noise. The 
FILTER? graph shows two distinct negative peaks at the points Barked. Tho 
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FIl.TF.R3 graph, on the other hand, has soeared the two peaks together In 
such n way that no pronrara could reasonably be expected to disambiguate 
then. Nevertheless, the relative magnitudes of the matched values In the 
three graphs are consistent with the two crossing interpretation indicated. 

5.4 DISCUSSION 

Flexibility as well as accuracy mist be regarded as keys to success 
with any automatic inspection system. This chapter describes an important 
stop forward in the dooain of castings inspection First, it has been 
dfmon&if ted that an automatic inspection system is capable of the accuracy 
required to implement the existing industry standard for grain size 
estimation. A certain amount of flexibility has also been demonstrated in 
the program's ability to maintain high noise immunity without a 
corresponding loss in resolution. 

Further flexibility requires that the measurements made in an lodge be 
related to object relief. This can be done tf the corresponding 
viowor-conterod representation of surface shape is available* A word of 
caution is in order, however. This discussion will describe how to moke 
the ncflsuremcnts required by ASTN standard E M2-74 on curved surfaces 
without establishing the validity of those mcaaurcnents. The goal here Is 
not to develop a statistical theory for estimating the three-dimensional 
grain size of oetals from arbitrary two-dimensional surfaces of 
observation. A5TH standard £ 112-74 develops the appropriate theory for 
planar surfaces or observation, and It will be left at that. 

Suppose one applies the three circle (Abrams) procedure to sections of 
an image of an curved surface of observation. A change in measured grain 
sizfi from one region of the image to another may be due to a change in 
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Figure 5-8 Throe circle (Abrams) procedure applied to macro-etched section 
or turbine airfoil. Figure 5-8(a) narks the grain crossings detected using 
a three concentric circle test pattern. Figure 5-8(b) shows the analysis 
or the intensity profile corresponding to the oiddlt circle of 
figure 5~B(a) t 
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actual grain size or to a change in the view angle. Determining actual 

groin size requires that the foreshortening effect of the view angle be 

explicitly accounted for at each image point. 

In a plan iine trie procedure, which involves counting the number of 

grains in a known area of test specimen, one can explicitly account for the 

effect of foreshortening if one knows the view angle (i.e., the oagnitude 

of the gradient) at each image point* The surface area corresponding to a 

tost region R in the image is given by: 

A » //sec(e) dxdy 
R 

where, once again, it Is assumed that the lnage Is an orthographic 

projection scaled so that image coordinates (x,y) correspond to object 

coordinates (x»y) 

In an intercept procedure, which involves counting the number of 

grains crossed by a one-dimensional curve on the test specimen, one can 

explicitly account for the effect of foreshortening if one knows both 

components of the gradient at each image point. The path length ds 

corresponding to a movement [dx,dy] in the image is given by: 



[dx dy] 
ds* = 



p 2 + | pq 
pq Q 7 *l 



(5-4.1) 



Equation (5.4.1) has the interpretation that the component of ds in 
the direction of steepest descent is foreshortened by the factor cos(e) 
while the component of ds in the direction of constant range is unchanged 
(see Appendix A. 5), 

Thus, in order to interpret the measurements derived by applying the 
three circle (Abrams) procedure to sections of an imago of an arbitrary 
curved surface of observation, it is necessary to have the viewer-centered 
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representation for surface shape developed in Chapter Z. Conversely, with 
such a representation, the interpretation becomes straightforward- 

The macro-etched turbine vane used in the example above was prepared 
using a contrast etch. A contrast etch makes the surface reflectance of 
the casting nonhomogencous. Thus, in order to determine the 
viewer-centered representation required by (5.4.1)* it would be necessary 
to first view an unetched part or have the representation supplied 
externally. On the other hand, for parts prepared using a flat etch, these 
two operations can be combined if the oethod for determining the 
viewer-centered representation is nodified to ignore the dark markings 
corresponding to the grain boundries. 

Fortunately. for airfoils, additional simplifications can be 
exploited* Airfoils can be approximated as singly curved surfaces. There 
is negligible curvature in directions parallel to the leading and trailing 
edges. Directions orthogonal to the leading and trailing edges are either 
strictly convex, on the "upper" surface, or strictly concave, on the 
"lower* surface. 

In this chapter, emphasis has been placed on a method for 
quantitatively estimating the average grain size of metals. In quality 
control, the requirements are often oore qualitative. It should be clear, 
however, that this chapter has developed a foundation for implementing a 
variety of qualitative techniques. The key Is to be able to relate 
measurements made in an image to the topography of the surface being 
viewed* Columnar grain can be detected by comparing grain size estimates 
using two sets of linear intercept procedures. One set would sample lines 
parallel to the leading and trailing edges. The other set would sample 
lines orthogonal to the leading and trailing edges. Any significant 
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discrepancy would indicato Lho presence or columnar grain, 

A more local version of the three circle (Abraras) procedure can be 
used to Asign a grain size estioate to each point in an image* Demarcation 
lines between regions of different grain size would then appear as boundary 
lines In this "image* of local grain size values* 

FOOTNOTE TO CHAPTER 5i 



1. ASTfl Designation: El 12 - 74 is equivalent to American 
National Standard Z30.9 of the American National 
Standards Institute and has been approved by tho 
Department of Defense to replace nethods 311.1 and 312 
of Federal Test flethod Standard 151b and for listing In 
the DoD Index of Specifications and Standards- 
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6. EXPERIMENTS RELATING REFLECTANCE DATA AND OBJECT RELIEF 
This chapter discusses the experimental determination of a roflectance map 
and its relation to object relief. The goal is to better understand how to 
acquire the data required to apply reflectance map techniques to practical 
situations. There is little literature on the measurement of the surface 
reflectance function #(i,e,g) required for Image analysis, A recont 
publication attempts to establish standards for nomenclature and 
measurement of reflectance [Nicodetmis r; at 77]. The results presented 
here, however, are meant to be illustrative, not conclusive. 

6.1 SPECIFYING SURFACE REFLECTANCE 

Two distinct physical processes are responsible for the reflection of 
radiant energy at boundary surfaces. The first of these is the mirror-like 
spcculor reflection. The second oT these is the matte reflection which is 
believed to occur due to multiple reflections at the boundaries of small 
particles of which the surface is composed. This second process will be 
referred to as diffuse reflection, Tho reJUcting power of a material Is 
defined by: 

la 

where I_ is the intensity of incident radiant energy and I Is the intensity 
of energy reflected by the medium. There is confusion over terminology. 
The convention is sometimes adopted that the reflecting power associated 
with a specular process is termed reflectivity, while the reflecting power 
associated with a diffuse process is termed reflectance 
[Uendlandt & Kecht 66]. Elsewhere, the term reflectance is used to cover 
both cases [rtorgan 53], [Jenkins & White 57]. Fortunately, In the 
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de term i nation of the reflectance nap R< P,q> t no fundamental distinction 
need be made between specular and diffuse reflection* In this work, the 
term reflectance is used to cover both cases. 

A conplete knowledge of the reflectance properties of a particular 
sample involves the determination of the spatial and spectral distribution 
of the reflected radiation with respect to both intensity and state of 
polarization. Reflectance ncasurements are an inportant tool in optics and 
analytic chemistry. The dependence of the reflecting powor of optically 
smooth materials on wavelength, polariz-ation and angle of incidence can be 
used to deternine the fundamental optical constants of the material. 
Reflectance spectroscopy extends this analysis to materials with nonsznooth 
microstructure. Such studies are useful for the analysis of powders of 
known particle size and shape ground fron satiples that cannot be analyzed 
using traditional spectroscopic techniques, and for chemical constituent 
analysis of compound substances [Wendlandt & Hocht 66]. But, reflecting 
power measures only the fraction of the incident intensity reflected and 
not its spatial distribution. 

Making use of reflectance measurements in image analysis requires an 
explicit account of the spatial distribution of the reflected energy as a 
function of the spatial distribution of the incident energy. It takes two 
parameters to specify a direction. But. if the reflecting material Is 
Isotropic, then the three photoaetric angles i. e and g. defined in 
figure 2-2. are sufficient to characterize the spatial relationship between 
Incident and reflected energy. Let Id be the intensity of incident radiant 
energy as before. Consider a surface element of size ds. The surface 
element ds reflects energy into the hemisphere defined by o ^ n/Z. Let I| 
bo the intensity reflected by the surface element ds in the direction of 



R. J, Woodhan 162 Relating Reflectance and Relief 

tho viewer measured per unit solid angle per unit surface area 
perpendicular to the emitted ray. Ii is called the radiance of the surface 
element ds and determines ho* bright it will appear when seen fron view 
angle e. The surface reflectance function *(i,e,g) Is defined by: 

*U,t a0 ) - -^- (6,1.1) 

Equation (6-1,1) does not distinguish between rotations of the surface 
eieraent ds about the surface normal (refer to figure 2-2). Such rotations 
do not change i f e or g. This is the sense in which the surface material 
Bust be isotropic. (6.1.1) does not include the dependence of surface 
reflectance on the wavelength of tho incident illumination. Thus, to use 
the corresponding reflectance map R(p,q) In lnage analysis. It Is necessary 
to determine *{l,e.g) under the sane illunination conditions and sensor 
tranxtor characteristics that will obtain when objects ar* to be viewed. 

One method for determining a reflectance sap which avoids theso 
problems is to oeasurc the intensities recorded in an image of an object of 
known shape. The reflectance nap R(p.q) so determined will hold for 
objects made from the same material and viewed with the same imaging device 
and under tho same object surface, light source and viewer geometry. 

Another method for determining a reflectance map is to use a 
photo-gon lometer to fix g and explicitly measure the dependence of 
reflectance on the two photometric angles i and e. Such measurements were 
carried out using an improvised photo-goniooeter end a sample gray iron 
casting as the specimen material. 
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65 MEASURING THE REFLECTANCE OF CAST CRAY IRON 

First, in measuring reflectance, it is important to be clear on what 
the moasurenents are measureoents of. Other definitions of surface 
reflectance arise which differ from the 6(i.o»g) defined In (6.1.1) by 
constant factors or by additional terms of cos(e) and/or cos(i)« These 
distinctions can be made clear if one carefully analyzes the ©easurement 
situation. 

Consider the experimental situation depicted in figure 6-1. Here, 
there Is a distant source producing a narrow colligated beam of incident 
illumination. Suppose the incident intensity is lo per unit area measured 
perpendicular to the incident bean* Suppose the cross-sectional area of 
the incident beam is dA. In this situation, the total radiant flux 
incident on the surface is equal to lodA, which is independent of 1. Let 
the corresponding area of surface illuminated by the incident beam bo ds. 
Then, dA corresponds to a foreshortened view of ds as seen by the source so 
that ds m cos<i)'dA. Suppose that the receptive field of the distant 
detector is large compared to ds so that all of the flux reflected in the 
direction of the viewer is captured by the detector. Let F be the flux 
measured by the detector and lot h douoto its intensity measured per unit 
solid angle per unit area perpendicular to the emitted bean. Then, 
F = l|cos(e)ds so that: 

#(i,«,g) = _!i_ c E = F C °M') (6.2.1) 

lo lo cos ( e )ds I ■ cos( e )dA 

(6.2 , 1 ) relates measurements obtained in the experimental situation 
depicted In Tigure 6-] lo the definition of the surface reflectance 
function given In (6.1.1). lo and dA are constant* Multiplying the 
detector measurement F by the corresponding cos( i)/cos(c) produces a 
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SOURCE 
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surface element 
di 



cross-section dA 



Figure 6*1 Measurement situation corresponding to a collimated beam of 
Incident illumination which is narrow compared to the receptive field of 
the detector* 




SOURCE 



o; 



surface elernen 
ds 



Figure 6-2 Measurement situation corresponding to a collimated beam of 
incident illumination which is wide compared to the receptive field of the 
detector. 
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quantity proportional to 6(1, c*g). 

Conpare this to the experimental situation depicted in figure 6-Z. In 
this second situation, there Is a distant source producing a colliraated 
beam of incident illumination that is wide compared to the receptive field 
of the distant detector* Suppose the cross-sectional area of the receptive 
field of the detector is dR and let t5s be the corresponding area of 
surface. Then, dR corresponds to a foreshortened view or ds as seen by the 
detector so that ds « cosfe^'dR. Let F be the flux measured by the 
detector and let Ij denote its intensity measured per unit solid angle per 
unit area perpendicular to the enittcd bean. Then, F a I|Cos(e)ds so that: 

*(l,o,g) = OL- . 1 . -?_ (6.Z.2) 

lo locos(e)ds lodR 

{6.2.2) relates measurements obtained in the experimental situation 

depleted in figure 6-2 directly to the definition t ! the surface 

reflectance function given in (6.1.1). Io and dR are constant. The 

detector measurement f is proportional to *(i,c,g). This definition of 

4(i, e f g) is the natural one to use in image analysis. Figure 0-2 correctly 

depicts what happens in an imaging situation. Picture elements, or pixels, 

in an image do not sample single points on the object surface. Rather, 

each pixel measures a flux which corresponds to a surface element ds 

foreshortened by the view angle e- This surface foreshortening makes each 

surface element appear brighter, in the image, as a function of cos(e). 

The experimental situation under which the photo-gonioueter 

measurements were made for cast gray iron corresponds to that depicted in 

figure 6-1* The particular goniometer geometry is shown in figure 6-3. A 

LEITZ PRADO UNIVERSAL projector was used as the light source. An opaque 

slide containing a small pinhole was projected to produce a collimated bean 
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or about 1 cm in diameter. A UNITED DETECTOR TECHNOLOGY. INC- 
SOX OPTO-HETER fitted with a tele-photometer was used as the detector. The 
instrunent was set to measure flux (in watts). At a distance of about 2 m w 
the receptive field oT the tele-photometer subtended about B cm of sample 
surface. A small planar section of specioen material, about 3 cm by 8 en. 
was mounted at the origin of the XVZ coordinate system. The phase angle g 
is the angle between the light source and the tele-photometer. For a 
particular set of cieasurements. tho phase angle g was fixed. The view 
angle e and the angle of Incidence 1 were varied by rotating the sample 
about the x-axls and y-axis. 

To generate a reflectance map, gradient coordinates p and q must be 
defined in terms of the goniometer geoaetry shown in figure 6*3. Two 
anomalies of the goniometer geometry must be dealt with. First, the 
positions of the source and viewer have been interchanged with respect to 
the standard imaging geometry used to define the reflectance map in 
Chapter 2.3. Vector [0,0,-1] points at the source while vector 
[sinfgj.O.-cosfg) J points at the viewer. Second, with the particular 
oechanical arrangement used, the x-axis and y-axls are coupled* That is, 
rotations about each axis are not independent. Let tf f bo tho Initial angle 
of rotation about the x-axis. Let ff ¥ be the subsequent angle of rotation 
about the y-axis. Specifying the order of rotation is necessary. For the 
results presented here, 6\ is set first followed by J y . Measurements were 
roade in increments of 5° in both 0, and fi v 

If one first rotates 0* about the x-axis and then rotates $ v about the 
y-axis, the equivalent gradient coordinates describing the resultant 
surface orientation are given by; 
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Figure 6-3 Photo-goniometer geooetry. 
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P = un(f v -a) 

q = tan<tf, ) 
cos(* tf -g) 

In this schetne, the position of the light source is given by: 

p G = -tan(g) 

qt » 
For the oeasurements obtained, the phase angle g was fixed at g = 35°* 
To determine the reflectance map required for linage analysis, the 
photo -goniometer measurements were multiplied by the "correction factor" 
cos(i)/cox(e). as indicated in (6,2.1), Subsequently, the measurements 
were normalized so that the maximun reflectance value was 1.0. The results 
are presented as a contour plot- Figure 6-4 plots these contours, spaced 
0.1 units apart. The box enclosing the contours dellaits that portion of 
gradient space which was measured. This limit reflects constraints icposed 
by the size of the specinen and the mechanical arrangement of the 
photo-gonioraetor. 

Reflection from the surface of a metal is highly specular. 5pecular 
reflection froa an optically smooth surface is easy to characterize. For 
such reflection, *{i,e.g) is zero except when f = ■■ and I. - and g lie in 
the sane plane. Then, *{i,e,g) is one. The viewing direction Tor which 
e = i and i, e and g lie in the same plane is called the perfect specular 
direction. As surface roughness increases, highly specular materials will 
reflect light in directions away froa the perfect specular direction. For 
any viewing direction ft. one can determine the angle between e and the 
perfect specular direction. This angle is called the ojj'speculurity 
angle 3. It can be shown that 

cos(s) = 2cos(i)cos(e) - cos{g) (6.2.3) 
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Ficiirr 6-4 Measured reflectance map for cast gray Iron with light source »t 
n, = -0.7 and Qi = 0.0 (contours are spaced 0.1 units apart). The solid 
box ■■rk* the region of gradient space over which neasurenents were made. 
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Figure 6*5 plots contours of constant cos(s) for p* = -tan(35°) q» = 0, 
spaced 0.1 units apart. Contours of constant cos{s) are circles in 
gradient space. In general, ir the light source is positioned at (pi,q»). 
then the contour cos(s) ■ kifl is a circle in gradient space centered at 
(cp$ ,cq* ) and of radius r where: 

c c co *<g> 
It + cos(g) 

and ^ B 1 - k 2 

[k + cos(g>]* 

For most surfaces, measurements suggest that both specular and diffuse 
reflection is always present, the relative proportion of each depending on 
the nature of the material [Wendlandt & Hectit 66]. By combining a tern in 
cos(s), to account for the specular component, and a tena in cos{i), to 
account for the diffuse component, it is possible to model different 
surface aatcriels. 

A good approximation for many materials is achieved by lotting the 
surface reflectance function be: 

*<i lC ,g) * t< n ** >«! («)■ ♦ (l-Dcos(i) (6,2.4) 

where t lies between and 1 and determines the fraction of Incident light 
reflected specularly, a paraneter which models the optical properties of 
the material, and n determines the sharpness of the specular peak, a 
parameter which models the surface nicrostructure [Horn 77 j - Figure 6-6 
plots the reflectance nap obtained using (6.2,4) with p* » -0.7, q. ■ 0, 
n = 3 and t = 0.6, There is good qualitative agreenent between figure 6-6 
and the data shown in figure 6-4. This agreement, however, has been 
verified for only one phase angle and over a limited region of gradient 
space* 
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Figure 6-6 Phenoaenologlcal model of reflectance nap for gray cast Iron with 
light source at p* = -0*7 and q« ■ 0.0 (contours are spaced 0.1 units 
apart) . 
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It is important to point nut that (6.2.4) Is a phcnonenological nodol 
and is not based on any optical theory or actual reflectance measurements. 
The purpose is not to produce closed form expressions for the surface 
reflectance of gray iron. At best, such an expression would vary with 
alloy composition and sand grain size. Rather, the purpose has been to 
demonstrate that a reflectance nap for d particular alloy with a particular 
surface nicrostructure can be determined empirically and Is, at least, 
consistent with phenomenological models of reflectance already in 
existence. 

The measurements required to deternine a reflectance nap are tioe 
consuming when bade Manually. A recent effort has been made to instrument 
the goniotaoter geometry illustrated in figure 6*3 so that reflectance 
measurements can be gathered automatically by computer [Aaoar 7BJ. 

Examination of a number of materials under a number of illuminations Is 
necessary to develop and evaluate phonoocnologleal node Is of reflectance. 

63 OBTAINING RELIEF DATA 

Object relief data was obtained for the shuttle eye of Chapter 4*2 
using an improvised coordinate gauging machine. A simple program was 
written to use a force controlled nechanical manipulator [Silver 73], A 
needle probe was inserted between the manipulator grippers and the surface 
scanned at a resolution of 0.01 inches in x and y. At each (x,y) point, the 
manipulator was programmed to descend until it "touched* the surface. The 
height of first touch was recorded to create a range file. Figure 6-7 
plots, from four different viewing directions, the range data obtained for 
the shuttle eye. Figure ti-B is an im-age of this shuttle eye synthesized 
from the range data and the reflectance map determined above. 
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Figure 6-7 Plot of measured terrain data for shuttle eye (from four views). 
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Figure 6-8 Synthesized image of shuttle eye using measured terrain data and 
phenomenological model of reflectance nap with light source at pi = and 

Qt ■ 0, 
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K CONCLUDING REMARKS 
This chapter summarizes the research in terms of the goal* initially sot 
forth and the results achieved. The work had two goals: one theoretical 
and one practical- The theoretical goal was to understand how to interpret 
image Intensity in terns or the underlying surface topography of the 
objects being imaged. The practical goal was to explore how machine vision 
systems could be applied to the problem of inspecting surface defects In 
metal castings. 

With these goals in mind, the research naturally split into two parts. 

The first part consisted of a basic investigation into the problem of 

determining object relief directly froo ioage intensity, laage analysis is 

a hard problem because many factors influence the intensity values recorded 

in an image. Image analysis requires that properties of image intensity be 

related to properties of the objects being imaged. This work has explored 

the fundamental nature of this computation. [Horn 75] [Horn 77) provided a 

framework for modeling both the geometry and radiometry of the 

image-forming process. A key observation is that, in a mathematical sense. 

tho problem of computing object relief locally at each image point is 

underdetermined. In order to assign a global interpretation to an image, 

it necessary to invoke additional as&uaptions about the nature of the 
object surface in view. 

The present work has extended the pioneer work of Horn by providing a 
mechanism Tor representing assumptions about surTace curvature and by 
demonstrating how this representation can be exploited in methods for 
computing object relief froo image intensity. The image Hessian has been 
defined and used to represent surface curvature. Assumptions about surface 
curvature map naturally into properties of the image Hessian, 
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In its theoretical contribution, the research is evolutionary rather 
thdii revolutionary. Dealing with arbitrary surfaces remains a difficult 
problem. In general, even if the reflectance map R(p.q) is known 
explicitly, there is no uniQue surface z = f(x,y) corresponding to a given 
image I(x,y) . Vet, people provide consistent Interpretations of images* 
One unexplored area of research is to try to fomulate constraints on the 
computation which lead to the solutions people most often assign. An idea, 
for example, is to compute the surface z = f{x,y) which requires the least 
surface area to account for the observed change In intensity. Such 
constraints, while interesting, fall outside the scope of the present work. 
In deternining what can be computed directly fron image intensity, it is 
also useful to point out what cannot. 

At the same tine, many examples have been presented In which image 
analysis simplifies. Surfaces which are singly curved and surfaces which 
are solids of revolution (i*e., have a circular cross-section) are 
particularly easy to deal with. Such simplifications arise naturally in 
practical situations since there are few design techniques and few 
fabrication methods for manufacturing parts with arbitrary doubly curved 
surfaces. Thit design of curved surfaces has always presented difficulties 
to the engineer- 5onc types of curved shapes, such as cylinders, spheres, 
surfaces of revolution and singly curved surfaces can be represented on a 
two-dimensional drawing with considerable precision. Wherever possible the 
engineer constrains curved portions of the design to one of these types * 
There are few drafting techniques for representing curves in a drawing 
which are not planar curves in space. Mot surprisingly. such 
simplifications also seem to arise naturally in the interpretation humans 
assign to drawings (ttarr 77a], 
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The second part consisted or a practical demonstration of a working 
inspection system. Having discussed the needs of casting Inspection with 
various representatives of the castings industry, a particular inspection 
problem was chosen. The industry standard for estimating the average grain 
size of metals was implenented and demonstrated on a sample turbine vane. 
Thp mage analysis portion of the program incorporated existing ideas in 
■achinc vision [flarr 76], [Lozano-Perez 77]. For a standard part surface 
and a standard method of part presentation, analysis of Intensity profiles 
is sufficient to verify grain structure for equiaxed turbine blades and 
vanes. This demonstrates that nachlne vision systems can tackle existing 
inspection problems. But, flexible inspection systems must be able to 
interpret measurements made from an image in terms of the underlying 
surface topography of the part being inspected. 

This example relates to an important fact of industrial automation. 
Current large volume automation incorporates a wide variety of special 
purpose 'tricks' to accooplish specific tasks or parts feeding, assembly 
and inspection. Unfortunately, there is little or no general theory of how 
to do this. This presents no problem when the volume of production 
Justifies an investnont in special purpose trickery, But. it also means 
that low to old volume production can not profitably be automated. 

rtachlno vision is an attractive interface to existing production 
facilities. In the immediate future, industrial applications of Dachine 
vision can be expected to take the form of special purpose tricks. This is 
reasonable if the donain of interest is constrained so that special purpose 
tricks will work. At the seme line, one would like to have an underlying 
theory of image formation. At tho very least, such a theory would help in 
the design of special purpose tricks. 
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Analyzing surface defects In green sand mold castings seems to require 
more than special purpose tricks. One idea night be to compare the lDage 
of each casting against the imago or a casting known to bo free of defects. 
Parts whose intensities natch to within a specified tolerance would be 
accepted while those outside the specified tolerance would be rejected. 
This idea is like a method of inspection used in many foundries. Castings 
can bo weighed cheaply. If the casting does not weigh as much as it 
should, then there must be shrinkage voids or regions of high porosity 
(i.e., holes) inside the casting. This Is a convenient way to find obvious 
internal defects without the high cost of a more quantitative analysis 
(I.e., radiographic or ultrasonic inspection). This method is successful 
since there is a simple linear relationship between measured weight and the 
mass of metal missing from the part. 

Unfortunately, in comparing images, there is no such simple 
relationship between differences in intensity and surface defects- A 
particular surface feature maps into many possible features of intensity 
depending on the position of the light source and and on the surface-viewer 
geometry. Simple difference detectors will fail unless the measurements 
made can be interpreted in the context or the part as a whole. 

One way to relate image intensity to part geometry is to match the 
image to a known model of the part. Two-dimensional models are useful for 
determining part identity, position and orientation from boundary 
information [Perkins 77J. Interpreting local features of intensity as 
defects on sections of smooth surface, however, requires a 
three-dimensional surface model. The surface function z ■ f(x,y) Is such a 
three-dimensional oodel. 
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For a surface with homogeneous optical properties, photometric stereo 
is a straightforward Method for iii'iu nnum the underlying surface 
2= f(x.y). This technique seems particularly suited to casting 
inspection- the two properties which deternine the surfaco photometry of a 
casting, namely alloy composition and surface nicrostructure, are carefully 
controlled. The production engineer is also free to standardize tho 
illumination and viewing geometry at each inspection station. 

Typical surface defects in castings (eg. pinholes, cold shuts, cracks 
and hot tears) manifest themselves as properties of surface topography that 
could not have been the result of an intended casting operation. To do the 
kind of first visual inspection required in a typical batch-orientod 
foundry, it is probably not necessary to have a precise model of ooch part 
geometry. Imagine an inspection system for finding cold shut defects in 
grcon sand nold castings* Photometric stereo could be used to determine 
the shape of the casting. Subsequent analysis would look Tor variations in 
intensity at concave boundaries between large sections of the casting that 
indicate a lapping or layering of the surface. 

This research has explored ways of determining the surface function 
z " f(x.y) directly froo image intensity. Exciting new techniques in image 
analysis are possible once the ioage I(x,y) has been aligned with the 
surface function z = f(x.y). Once alignment has been achieved, it becomes 
possible to distinguish effects due to illunination and varying surface 
Slope from effects due to varying surface cover [Nitzan et irj 77] 
[Barrow A Tcnonbaura 78] [Horn & Bachnan 77] 
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APPENDIX A: MATHEMATICAL DETAILS 

11m purpose or this appendix is to explore more formally the question of 

how physical constraints on the object surface z = f(x*y) constrain 

possible solutions to the basic lisage-formlnp equation I(x t y) = R(p,q). 

Intuitive reasoning about smooth surfaces and gradient space needs to be 

augmented by some formal definitions and theorems. The mathematics 

presented in this appendix is not new* The pertinent definitions and 

theorems arise from work in linear algebra, convex analysis, differential 

geometry and nonlinear programming* The goal here is to apply this 

mathematics to the problem of interpreting image Intensities. The required 

formal results are merely summarized* Technical details of proofs havo 

been omitted* For those details. the reader is referred to 

[Mangasanen 09], [Moore 68], [Lucnberger 73] and IKepr 69]. 

In order to develop the required theorems, the notion of surface 

smoothness nust bo made precise* 

Definition. Let f(x) be a real-valued function defined on an 
open set r c R" . Then the (hyper)surface described by the 
equation z o f(x) is smooth over V if f(x) is twice 
dirrerentiable with continuous second partial derivatives 
at every x c T. 

It is also important to deal explicitly with surface curvature. The 

Hnssian matrix H is the generalization to R n of the concept of tho 

curvature of a function of a real variable* 

Definition. Let f(x) be a real-valued function defined on an 
open set r c FT and let x i I\ Then, the n x n matrix 
H = V 7 f(x*K whose ij 1h element is given by: 

eta* dXj 
is called the Hessian (matrix) of f(x) at X*. 
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The following theorem establishes that the Hessian matrix Is syimetrlc 

Tor smooth surface*. 

Theorem A J Let f(x) be a real -valuer! function defined on 
an open set T c R n and let x" < I\ If the (hyper)surface 
described by the equation z ■ f(x) is smooth at x , then 

j jf(x') * a_ af(x') 

3x* dxj 3xj £xi 



Corollary A.I If z « f(x) is smooth at x , then the Hessi 



an 



matrix H exists at x and is symmetric. 
It can now be shown that the Hessian matrix H underlies the ctodel of 
the image forming process. Once again, recall that the assumptions used 
are that the viewer is distant, that the image projection U orthographic 
taking object point (x,y,z) onto image point (x t y) and that each object 
point receives the sane incident illumination. The basic image- forming 
equation I|x,y) = K(p.q) is one equation in the two unknowns p and q. By 
taking partial derivatives of this equation with respect to X and Y, two 
equations are obtained: 

I- = PiRp + qtRa 

(subscripts are used to denote partial differentiation). Theorem A, 1 says 
that, for a smooth surface, p., = q, . Thus, two equations are obtained In 
the three unknowns p* , q, and p v = q, where: 



ax 7 ay 1 dx3y 



These two equations can be written as the single matrix equation: 



U p* to R* 

V Pv flv. Re 

The relationship between the vector of first partial derivatives of the 

intensity function [I B »I T ] and the corresponding vector of first partial 
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derivatives of the reflectance map function [R*.fU] is given by: 

IU.UT = H [Rp.Rqr (A.l) 

whnre H = V 3 f(x,y) is the Hessian matrix of second partial derivatives of 
the object surface z * f(x.y). 

First-order approximations, written in terms of differentials, 
relating a snail movement [dx.dy] in the image to the corresponding 
moveaent [dp.dqj in gradient space are given by: 

dp = p.dx ♦ p v dy 
dq = q, dx ♦ q,dy 
Ayain, these two equations can bo written as the single oatrix 
equation: 



Pv 



dy 



To a first approximation, the relationship between a snail movement 
[dx.dy] in the imago and the corresponding movement [dp.dq] in gradient 
space is also determined by the Hessian matrix M. It Is given by the 
equation: 

[dp.dqr * M (dx.dyf (A.Z) 

A few words of caution are in order. While equation (A.l) above is 
exact at any image point (x,y) and its corresponding gradient point (p.q), 
equation (A. 2) is only approximate. The Hessian matrix H Is a function of 
x and y. In the subsequent analysis, however, it is assumed that [dx,dy] 
can be chosen small enough so that H can be considered constant over the 
interval (x,y) to (x*dx,y+dy). 
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Suppose inngfi point (x # y) is known to correspond to gradient point 
(p.q). If two linearly independent directions [dxi,dyi) and [dxj.dyp] and 
tho corresponding f dpi f dqi ] and [dp?.dq?] are known, then the Hessian 
matrix H is determined uniquely at (x,y). Indeed, 



H 



dp> dp* 




dXj dx? 


dqi dq? 




dyi dyj 



-I 



A.I HORNS METHOD FOR OBTAINING SHAPE FROM SHADING INFORMATION 

It is now possible to re-exanine the aethod Horn developed for 

obtaining shape from shading information [Horn 75] [Horn 77], The goal Is 

to interpret Horn's oethod in terns of the image Hessian II. 

Tho basic recipe for Horn's solution is as follows: 

(i) Suppose image point (x ( y) Is known to correspond to a 
point {p.q) in gradient space. Then, the change In 
z ■ f(x ( y) corresponding to a small movement tdx,dyj 
in the image is given by the first-order 
approximation: 

dz = pdx * qdy 

(ii) The new gradient point corresponding to the image 
point (x*dx,y*dy> 15 obtained by updating the current 
gradient (p.q) according to the equation: 
[dp,dq] T = H [dx.dy] T 

Unfortunately, there is not enough information to determine the oatrlx 

H- The only constraint on H is that It satisfies the equation: 

th*hT s " [Ro.R-] T 
Note, however, that matrix ntultiplicatton is a linear operation. If 
[dx.dy] is chosen to be in the direction of [Ro.R«] then linearity Is 
sufficient to guarantee that [dp.dq] will be in the direction of [I*,!*]. 
Hore precisely: 

If [dx.dy] * [Bp.Ro ]ds ( then [dp,dq] a [ I, , I v ]ds 
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Thus, by starting at a point (x.y) Known to have gradient (p.q) and 
iterating the above two operations* a path in the image is traced out for 
which the corresponding gradients, and hence the corresponding relief 
profile on the object surface, can be determined. The catch is that an 
arbitrary direction for [dx,dy] cannot be chosen, [dx»dyj must be chosen 
in the direction [Ro.Ro]. The curves traced out on the surface in this 
fiishion are called cftarocterisitcs and their projections in the laage plane 
are called hasv characteristics . 

This result can also he interpreted geometrically. Choosing [dx.dy] 
to be in the direction (Rp.Rq] means that a base characteristic Is traced 
out that is always perpendicular to the contour or constant reflectance at 
the current (p.q). Similarly, the fact that the resulting [dp»dq] is in 
the direction [I, t I v ] neans that the corresponding path traced out in 
gradient space is always perpendicular to the contour of constant intensity 
at the current (x,y). Figure A-l illustrates this result. 

The path the base characteristic traces out cannot be controlled. It 
riepnnds on the particular object in view. Assumptions are required to link 
the characteristics traced out from different starting points, 

A.2 THE IMAGE HESSIAN MATRIX 

In this work, the requirement that [dx.dy] be chosen in a particular 
direction is relaxed. The reason Is not to make the problem more 
difficult! Rather, it Is to try to gain oore control over the "path" that 
can be analyzed, independent of the object in view. To do this* the 
analysis of how properties of the object surface z = f(x*y) relate to 
properties of the image Hessian natrix II is extended* These properties of 
H constrain the set of [dp,dq]'s that can correspond to a particular 
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Figure A-l Suppose image point (x,y) 
expanding a characteristic, movement 
normal to the contour or constant 
movement in gradient space is in the 
constant intensity at (x,y). 



corresponds to gradient (p.q) . In 
in the image is in the direction 
reflectance at (p,q). Similarly, 
direction normal to the contour of 
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* 

[dx,dy]. 

If II were known everywhere then the problem of obtaining shape from 
shading would be solved since Morn's method could be used to trace out an 
arbitrary base characteristic, independent of the object in view. On the 
other hand, if constraints on the object surface z = f(x,y) can bo 
expressed in terms of properties of H. then these constraints can be 
applied to the set of possible solutions to the basic image- forning 
equation I(x.y) ■ R(p.q). The Hessian matrix II is the generalization to 
R" of thn concept of curvature of a function of a real variable- Here. It 
is established that the corresponding positive (negative) definiteness of 
the Hessian is the generalization of positive (negative) curvature- 
Definition. Let A be a real n x n symmetric matrix. Then A 
is called positive semdejtnite if x T Ax * for all 
x c R ft ♦ A is called pMittH definite if x T Ax > for 
all nonzero x < R* - Similarly, A is called Motive 
*emide/inite if x T Ax < for all x < R n and A is called 
negative definite if x T Ax < for all nonzero x c R 1 * . 

H is a real syraietrlc oatrlx (Corollary A.I). Therefore, It 1ft 

appropriate to examine conditions under which H is positive (negative) 

definite. The positive (negative) definiteness of the Hessian matrix H is 

related to the convexity (concavity) of the corresponding (hyper)surface 

z = f(x). The following definitions and theorems establish the required 

results: 

Definition. A real-valued function f(x) defined on a convex 
set r c R" is said to be convex if* for every xi,X2 < T 
and every x, s a si, 

fUx» + (1 - AJx?) s xf(xi) ♦ (1 - X)f(xz), 
If. for every < X < 1 and Xi * X2 

fOXl + (1 - A)X2) < Xf(Xl) * (i - *)f(X?), 
then f(x) is said to be strictly convex. 

Similarly, a real-valued function f(x) defined on a convex 
set T c IR ft is said to be concave if -f(x) Is convex and 
strtctly concave if -f(x) is strictly convex. 
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Theorem A.2 Let f(x) be a real-valued function defined on 
an open convex set F c PT and let the (hyper)surface 
described by the equation z = f(x) be smooth over P. Then. 
f(x) is convex on T if and only if H a V z t{%) is positive 
semidefinite on I\ Similarly, r(x) is concave on T if and 
only if II = V 2 f(x) is negative scaldefinite on I\ 

Untortunately, theorem A.Z does not extend to strictly convex and 

strictly concave functions by simply replacing the inequalities by strict 

inequalities. The extent by which it does extend is given by the following 

theorem. 

Theorem A J Let f(x) be a real-valued function defined on 
an open convex set r c FT and let the (hypor)surface 
described by the equation z= T(x) be smooth over I\ A 
sufficient but not necessary condition that f(x ) be 

strictly convex on T is that H ■ V ? f(x) is positive 
definite on I\ Similarly, a sufficient but not necessary 
condition that f(x) be strictly concave on T is that 
II ■ V 2 f(x) is negative definite on I\ 

These results are used to show how convexity adds constraint. Suppose 
the surface z = f(x,y) is convex* Multiplying the two equations 
[I„.I V ] T = H [R ,Ra] T and [dp.dq] T = H |dx,dy) T on the left by [R*,Kq] and 
[dx,dy] respectively generates the two Inequalities: 

[Rp.Rnl II [R*,Ro] T ■ Rol. + RqIv * 

[dx,dyj It (dx.dy] T = dxdp ♦ dydq i 
The first inequality K„] a * R^ I ¥ > can be viewed as additional 
constraint on the contour in gradient space of possiblo solutions to the 
basic image-forming equation I(x f y) « R(p,q). Suppose Image point (xo,yo) 
has I(xo.yo) « a. Figure A-Z(a) shows the point (xo.yo) along with the 
corresponding image intensity contour I(x ( y) « a* The vector [I«,Iyl 
defines the direction normal to the contour I(x f y) a a at each point (x.y)* 
Figure A-2(b) shows tho reflectance map contour R(p.q) = o\ The vector 
[Rp*Ral defines the direction normal to the contour R(p,q) « a at each 
point (p.q). 
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k 



R(p,q) = « 




Vk^Vk * ° 



< ) 



(b> 



Figure A-? The inequality U-M.H restricts tn. contour in gradient 
space that can correspond to a given Wage point. 
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Rpli ♦ ***U > if and only if the angle between tho normal [I,,I V ] 
<ind the normal [Rp.R«] is less than or equal to 90°. Thus, if <xo,yo) lies 
on a section or surface known to be convex, points on the contour 
R(p,q) = a for which R»I a * H« i ¥ < o {the dotted section of the contour 
K(P.Q) = cr of figure A-2(b)l can be excluded fron the set of possible 
gradient points corresponding to inage point (xo.yo). 

The second inequality helps to constrain the movement in gradient 
SpftCt [dp.dq] that can correspond to a movement [dx.dyj in tho image. 
Suppose image point (xo.yo) is known to correspond to the gradient point 
(po.qo). The vector [dx.dy] defines the direction of movement in imago 
space (figure A-3(a>)* Now, dxdp ♦ dydq >0 if and only if the angle 
between [dx.dy] and tho corresponding [dp.diij is less than or equal to 90°. 
Thus, if a movement is made in the direction ldx,dy] from the inago point 
(*o.yo) on a section of surface known to be convex, then the corresponding) 
movement in gradient space must take tho point (po.qo) into the region R of 
figure A-3(b>. 

This second result can be used to choose [dx.dy] in such a way as to 
guarantee that tho view angle increases or that the direction of steepest 
descent increases. Figure A-4. illustrates what happens if [dx.dy] is 
chosen to be in the direction [po»«]. In this case, the line 
dxdp * dydq = is the tangent line to the gradient space circle 
P* * n ? ■ Po ? * qo*. All points in R now lie outside this circle so that 
tho point (po.qo) must move to a point of increasing view angle. 

Figure A-5 t illustrates what happens if [dx.dy] is chosen to bo in the 
direction [-qo.po]* In this case, thu line dxdp + dydq « is the lino 
connecting (po,qo> and tho origin (0,0). All points in H lie above this 
line so that the point (po,qo) must move to a point of increasing direction 
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of steepest descent. 

A3 A GEOMETRIC INTERPRETATION OF MULTIPLICATION BY THE IMAGE 

HESSIAN 

A usoful geometric interpretation of multiplication by a real 

synmetric matrix derives from the Tact that any positive definite matrix 

can be used to define a norm. This provides a geometric Interpretation of 

multiplication by a positive definite H* The interpretation can be 

extended to arbitrary II* 

Definition. Let A be an n x n positive definite natrix and 
let x c R" . Then. IxIa is called the d-Jiom 0/ x where 

IxU 7 ■ x T Ax 

The key observation is that an A-noro is like the standard Euclidean 

norm except that it can apply different weights to the components of x in 

different directions. These weights and directions are determined by the 

eigenvalues and eigenvectors of A* The following three theorems formalize 

this observation. 

Theorem A.4 Let A be a real synmetric n x n matrix. Then 
all the eigenvalues of A are real and there exist n 
mutually orthogonal eigenvectors corresponding to each of 
the (not necessarily distinct) eigenvalues of A, 

Theorem A.5 Let A be positive definite . Then all the 
eigenvalues of A are positive* 

Theorem A.6 Let A be a positive definite n x n matrix* 
Let X| .^a X« be the n positive but not necessarily 

distinct eigenvalues of A and let ui .<*>? wn be a 

corresponding set of n wutua lly orthogonal eigenvectors* 

(Without loss of generality, choose each &>» to be a unit 

vector and order the u, so that the vectors m ,w? ft** 

form a right-handed coordinate system*) Then* using 

wi.wz ua as a sot of basis vectors, any n-vector x 

can be rewritten in the form: 

x >yiwi ♦ yzw + ... + y*u* 

Then: 

IxHa 2 - Xiyi 2 + *iyz z + ... + A*y* 2 
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Y 

A 



(xQ*dw.)-Q*dy) 
(Xq.^q) 




I ) 



dpdx*dqdy=0 



I ) 



Figure A-i The Inequality dpdx 
space that can correspond to < 



y dpdy > restricts tho movcocnt In gradient 
given movenent In image space. 
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4 



(x *dx.J*o+dV) 



(o) 




Figure A-4 Surface convexity can be used to choose a ooveoent [dx.dy] in tho 
laage such that the corresponding niovenent [dp.dq] in gradient space 

incraascs the view angla a. 
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(xq-xq) 



Co) 




'dpdx + dqd^O 



(b) 



Figure A-5 Surface convexity can be u*cd to choose a movement [dx.dy] in the 
imago such that the corresponding movement [dp,dq] In gradient space 
increases the direction of steepest descent. 
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Theorem A. 6 follows from the fact that any n x n symmetric matrix A 
is orthogonally (rotationally) similar to an n x n diagonal matrix whose 
diagonal oleiacnts are the eigenvalues of A , Knowing that all the 
eigenvalue* of a positive definite matrix are positive guarantees that the 
quadratic form x T Ax satisfies the requirenents of a metric norm. 
Theorem A. 6 provides a useful geometric interpretation for multiplication 
of a vector by a positive definite matrix. To avoid unnecessary 
complication, let us specialize the Interpretation to the case of the 2 x Z 
Hessian matrix H corresponding to a surface 2 * f(x,y). Recall, from 
(A.Z) above, that 

[dp,dq] T = IUdx.dy] T 
For notatlonal convenience let 

a « p* . b » q y and c = p, * q. 
Then, it can be shown that the two eigenvalues of H are: 



M 


. a*b 
2 

. a*b 
Z 


2 


■b)< 


♦ 4c' 


xa 


Z 


-b)< 


♦ 4c* 



and that the corresponding (unit) eigenvectors are: 

ui = (cos(*h-sln(#)] 

wi ■ [sin(*).cos(*)] 
where 

tan(2») = -?£- 
b-a 

Multiplication by H can then be interpreted as follows; Idp.dq] is the 

vector sum of the projection of [tlx,dy] in each of the eigenvector 

directions ui and w? where each projection is scaled by the corresponding 

eigenvalues A» and A?. That is f [dp, dq] = \\ (v T wi im ♦ >2(v T <#>2 )&>2 where 
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v T * tdx.dy]. 

Consider moving 3 snail distance ds in the imago. Consider the family 

of all fdx.dyj such that ds = 7dx 2 + dy* ■ If II is positive definite at 

an image point (xo,yo) known to correspond to the gradient point (po.qo). 

then multiplication by II induces a 1-1 continuous napping of the image 

space circle centered at (xo.yo) and with radius ds onto the gradient space 

ellipse centered at (pD,qo) and with axes Aids and X?ds in the directions 

wi and u? respectively. Figure A-6 illustrates this result, with Image 

space coordinates and gradient space coordinates superimposed. The fact to 

be exploited is that the mapping from this image space circle to the 

corresponding gradient space ellipse is continuous and 1-1. Multiplication 

by a positive definite H is monotonlc in the following sense: 

multiplication by a positive definite H preserves the ordering in angular 

position of the [dp,dq] a & corresponding to given [dx,dy]'s. Let us make 

this more precise. 

Definition. Suppose two nonzero vectors x * [xi,X2] and 
y = [yi-Ste] * r * described in a right-handed (positive) 
coordinate system. Then x is said to be (strictly) less 
in angular position than y if the angle required to align 
x with f by rotating x in a counter-clockwise direction is 
(strictly) less than the angle required to align x with y 
by rotating x in a clockwise direction. Similarly, x is 
said to be (strictly) greater in angular position than y 
if the angle required to align x with y by rotating x in a 
counter-clockwise direction is (strictly) greater than the 
angle required to align x with y by rotating x in an 
clockwise direction. 

Definition. Let z = f(x,y) be the equation describing a 
smooth surface and let > ( and X? be the two eigenvalues of 
the corresponding Hessian matrix H at image point (xo,yo)* 
The surface z ■ f(x.y) is said to be planar at (xo*Uo) If 
and only if *i = A? = 0. The surface z » f(x,y) is said to 
he singly curved at <xo*yo) if and only if one (but not 
both) of Xi and X? is equal to zero. The surface 
z ■ f<x,y) is said to be doubly curved at (xo.yo) If and 
only if both Xi and >? are not equal to zero. 
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Figure A-6 Multiplication by the ionge Hessian H induces a 1-1 Mapping 
between a circle of radius ds in the inane and an ellipse in gradient 
space- The major and minor axes of this ellipse are determined by the 
eigenvalues and eigenvectors or II. 
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Theorem A.7 Let z = f(x.y) be the equation describing a 
smooth surface* Then z a f(x,yj is doubly curved at 
(xo.yo) if and only if the corresponding Hessian natrix H 
is nonsinguiar at (xo.yo). 

Theorem A.8 Let z = f(x»y) be the equation describing a 

smooth surface and suppose z = f(x,y) is doubly curved at 

image point (xo*yo)> Let H be the corresponding Hessian 

matrix at (xo*yo) having nonzero eigenvalues > and *2, 

(i) If H is positive or negative definite (\i and Xj have 

the same sign), then nultiplication by H preserves 

the ordering of angular positions of [dp v dq]'s Kith 

respect to the corresponding [dx«dy]'$* That is, if 

[dPi,dq,] T = H [dxudyif. [o>2.dqz] T = H [dxj^dyeT 

an 1 (dxi.dyi] is (strictly) less in angular position 

than [dx? t dyj] then [dpi f dqi J is (strictly) less In 

angular position than [dp2.dqz]* 

(ii) If H is neither positive nor negative definite (*i 

and >2 have opposite sign), then multiplication by H 

reverses the ordering of angular positions of 

[dp.dqVs with respect to the corresponding Idx.dyl's. 

That is. if 

[dpi.dqi] T » H [dxi,dyi] T , [dp2,dq 2 f = II [dx*,dy*) T 
and [dxj,dyi] is (strictly) less in angular position 
than [dX2,dy?l then [dpudqi] is (strictly) greater in 
angular position than [dp2,dqz]. 

Theorem A-8 requires that z = f(x.y) be doubly curved In order to 

guarantee that multiplication by H Is a 1-1 mapping- However, if H is 

nonsinguiar, then theorem A. 8 gives precise conditions with which to order 

the angular changes to position in gradient space corresponding to given 

movements in image space- If H is cither positive or negative definite 

then the mapping of wage space circle to gradient space ellipse goes from 

a right-handed (positive) coordinate representation to a right-handed 

(positive) coordinate representation so that the ordering of angular 

positions is preserved- If H is neither positive nor negative definite 

then the mapping of itiage space circle to gradient space ellipse goes froo 

a r 1 tjht-handed (positive) coordinate representation to a left-handed 

(negative) coordinate representation so that the ordering of angular 

positions is reversed. 
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A. 4 THE IMAGING MATHEMATICS OF A SPHERE 

Expressions can be developed for the surface orientation and the image 
Hessian matrix II for an inage of a sphere. The purpose Is not to perform 
an exercise in differential calculus. Rather, the analytic results 
developed here can be used to illustrate the connection between a 
viewer-centered definition of the image Hessian matrix H and the more 
traditional object-centered definition of curvature. 

Consider a sphere of radius r centered at the object space origin- 
The surface of the sphere is described implicitly by the equation; 

F(x,y.z) *x**y*+z 2 -r z =0 
This equation gives no indication of which points on the surface project to 
points in the image nor which points on the surrace are hidden from view. 
It is an object-centered representation of the surface. For the work here, 
an explicit representation is required- For a sphere, this explicit 
representation is given by the equation: 

* = f(x,y) > -/r 7 - x 2 - y* 
An explicit representation or the fom z = f(x.y) is a viewer-centered 
representation of the object surface. Recall, fro« figure 2-3. that the 
viewer is looking along the positive z-axis so that the points on the 
sphere actually in view correspond to negative values of z as indicated 
above. 

The gradient coordinates p and q are determined by differentiating 
f(x.y) with respect to x and y. One finds: 

4x 2 

a , dti*,y) c IX 

dy z 

Taking second partial derivatives of f(x,y) with respect to x and y, one 
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finds; 



p. = £m*Xl , .(r'-y 2 ) 
3x ! Z 3 

„ v = 3'f<x.v) B _ tr g -x z > 
ay* z 1 

p v = q , . ^f(x,y) E -H 
a«3y z 9 

Thus, the Hessian natrlx H Is given by: 

r ! -y2 xy 



H = -1/2 3 
Tho ngonvdlues of the Hessian natrlx II arc given by; 



xy r 2 -x 2 



ft. *=4 



*.=* 



with corresponding (unit) eigenvectors: 

tfl * (sin(tf) f -cos{fl)] 

w? ■ tcos(*),sin{»)] 

whore 

tan{«) = * 
x 

First, consider the special case x - o y - Then z ■ f (0 , 0) 

p » 0, q = and the Hessian matrix II becomes: 



= -i\ 



ii 



Mr 
1/r 

Thi* is as expected. The curvature in each or the two principal directions 
of movement Is the sane, and is equal to one over the radius of the sphere. 
This is actually true for any point on the surface of a sphere. 
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{ Conversely, a sphere 1$ the only surface which has constant nonzero 
curvature in All directions and at all points.) But. this notion of 
surface curvature corresponds to an object-centered representation. The 
imago Hessian matrix H is not constant for all (x ( y). 

A.5 RELATING THE IMAGE HESSIAN TO SURFACE CURVATURE 

The way the image Hessian matrix II has boon doflnod In this report 
corresponds to a viewer-centered representation of curvature. The Hessian 
matrix H relates Doveinent in the image to changes in local surface 
orientation and not movencnt on the object surface to changes in local 
surface orientation. The case x = 8 y= 0, cited for the sphere above, is 
the unique situation for which the object-centered definition of curvature 
and the viewer-centered definition of curvature coincide. Here, the 
Hessian matrix H intuitively captures what one would expect from a 
"curvaturo* matrix. In a viewer-centered representation, the ioage Hessian 
matrix H of a sphere is not constant- The dependence of H on x and y 
captures the variation in apparent curvature when the surface is viewed 
obliquely. 

This is oado clear if the expression for the image Hessian H of a 
sphere is rewritten in terns of gradient coordinates p and q One finds: 



H = l/cos(e) 



P 2 +l PQ 
pq q 2 +l 
where o is the view angle. That is: 



1/r 
l/r 



(A. 3) 



cos(e) = 



1 



A + P* ♦ q 3 
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The following two theorems provide an interpretation for (A. 3) above: 

Theorem A,9 Let z = f(x t y) bo the equation describing a 
smooth surface. Let dA be a differential olenent of area 
on the surface. Then, 

dA m i dxdy 

cos(e) 
where dxdy is the corresponding differential element of 
area in the Image and e is the view angle subtended by the 
surface element dA. Thus, the surface area corresponding 
to a region R in the inage is given by; 

A c //sec(o) dxdy 
R 

Theorem A.IO Let 2 - f(x.y) be the equation describing a 
smooth surface. Let ds be the differential of the arc on 
the surface corresponding to a movement [dx.dy] in the 
image* Then, ds is given by the A-norm of [dx.dy]. That 

ds * B[ dx.dy ]h = A dx.dy JA[dx.dy] T 
where the natrix A is given by: 

P 7 *! pq 
A = 

pq q 2 +l 

Theoreo A. 9 allows one to interpret multiplication by l/cos(e) In 
(A. 3) as compensation for the foreshortening of area due to the oblique 
view corresponding to a gradient (p,q). Theorem A.IO allows one to 
interpret multiplication by A in (A* 3 J as compensation for the 
foreshortening of path length due to the oblique view corresponding) to a 
gradient tp.q). Finding the area of surface corresponding to a given 
region of inage depends only on the magnitude of the gradient at each image 
point in the region. On the other hand, finding the path length along the 
surface corresponding to a given curve In the image depends on both the 
magnitude and angular position of the gradient at each image point on the 
curve. 

This dependence of path length on the gradient (p,q) can be made oore 
explicit by, once again, examining the eigenvalue and eigenvector structure 
of A. The matrix A is positive definite with eigenvalues: 
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Xi = 1 



Xp= l 



cos*(e) 
and corresponding (unit) eigenvectors: 

m = [sin(*),-cas(*>J 
u? * [cos(ff).sin(ff>] 

whore 

tan(l) = 3 
P 

The component of the differential ds of the arc on the surface in the 
direction of steepest descent is foreshortened by the factor cos(e) while 
the component of the differential ds in the direction of the contour of 
constant z = f(x.y) is unchanged- 

Since A is positive definite, it is also invertible and its inverse 
A" l Is positive definite. Knowing the gradient point (p,q) and the 
Hessian matrix H at an image point (x.y) allows one to determine the 
magnitude and direction of the object-centered principal radii of curvature 
or the surface z = f(x t y). Suppose that k\ end *2 are the two eigenvalues 
and ua and u? are the corresponding unit eigenvectors of the matrix C 
where: 



C « cos(e>A' 'H = cos 3 <e> 



q* + l -PH 
-pq p*+i 



H 



Then, n = 1/Ki and rz » l/k2 are the two principal radii of curvature of 
the surface z = f(x,y) oriented respectively in the directions derined by 

tfi and 02 » 
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Finally, since cos(e) > and A* 1 is positive definite, one observes 
that the number of nonzero eigenvalues of H is equal to the number of 
nun/oro eigenvalues of C = cos(ft)A +l H. An eigenvalue A, of H is zero 
(nonzero) precisely as the corresponding principal radius of curvature r* 
is infinite (finite). Thus, the definitions of a surface z * f(x.y) being 
planar at (xo.yo). singly curved at Ixo.yo) and doubly curved at (xo.yo) 
given in terns of the viewer-centered Hessian matrix H are equivalent to 
the nore standard definitions given in terns of the object-centered 
principal radii of curvature n * l/k| and r* = 1/k?. 
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APPENDIX B: CATALOGING CASTING DEFECTS 
The initial goal in exploring casting inspection as a suitable area of 
application for machine vision was to develop a sinple catalog of defects 
in octal castings. Tho first observation to be made is that the phrase 
"defects in oetal castings" encoopasses a broad spectrum of possible 
issues. The defects associated with a particular casting process fall 
roughly into one of three categories: 

1 , INHERENT DEFECTS 

- defects introduced during the preparation of the aetal 
alloy or other rax naterlal* 

2, PROCESSING DEFECTS 

- defects introduced during the casting process 

3, SERVICE DEFECTS 

- defects introduced during the operating cycle of the 
casting 

Here, only defects introduced by the casting process Itself are 

considered* It Is appropriate to further split such processing defects 

Into three categories; 

1. SHAPE DEFECTS 

- defects that affect the overall di»ensional accuracy 

of the part 

Z. SURFACE DEFECTS 

- defects that manifest themselves as local surface 
properties of the part 

3, STRUCTURE DEFECTS 

* defects that affect the nechanical response of the 
part 

These three categories of processing defects are not mutually 

exclusive. There is considerable overlap between categories. The kind of 

inspection considered in this work rclatos primarily to SURFACE defects. 

SHAPE defects related to the accurate verification of part di&ensions are 

excluded. STRUCTURE defects which can be found only by destructive 
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testing, special test equlpoent or by inaging rays which penetrate the 
object surface (eg. X-rays, ultra-sonic rays) are also excluded. As the 
catalog below demonstrates, many defects which arfect the structural 
properties of a casting are nonetheless manifest as surface properties. 
These are not excluded. 

Before presenting the catalog, a slight disclaimer is in order: this 
catalog of casting defects really should take into account the particular 
alloy used, the particular casting technique chosen and the particular part 
geometry. Fortunately, however, each alloy, each casting technique and 
each pari geometry share the sane broad categories of defects. The 
difference lies in the relative rate of occurrence of the various kinds of 
defects and, to a lesser extent, in the way in which these defects manifest 
themselves. 

With that disclaimer, here is the catalog of defects: 

COLD SHUT <FLOW HARM 

DESCRIPTION: 

A defect that affects both the SURFACE and the STRUCTURE of a casting. 

Broadly speaking, a cold shut defect arises when molten octal does not weld 

together properly. There are two types of cold shut defect. One type 

occurs at the interface of two streams of molten metal whose temperature 

differential prevents then from welding together properly. This is 

manifest as a lapping or layering on the surface of the casting* The other 

type results from loose droplets of molten oetal entering the mold cavity 

ahead of the aain streao of oetal. These solidify and become partially 

enhedded In the surface of the casting. 

CAUSES 
DESIGN: 

- portions of the mold too cold 
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PRODUCTION: 

- pouring/injection rate too slow 

- mold temperature loo low 

- metal temperature too low 

- dirty metal (gating blockages) 

CRACKS 

DESCRIPTION: 

A defect that affects both the SURFACE and the STRUCTURE of a casting. A 

crack is a defect that occurs during contraction shrinkage following 

solidification. Localized stresses aro sot up within a casting immediately 

following solidification, especially at junctions of restraining ribs, 

right angle intersections, and junctions between thick and thin sections, 

Those stresses can produce cracks as the casting cools to roo« tenperature. 

Because such cracks occur after solidification, there is little chance for 

oxidization and they are typically very clean. 

CAUSES 
DESIGN: 

- uneven or too rapid cooling 

- excessive mold rigidity restraining normal contraction of the 

metal 
PRODUCTION: 

- mechanical jarring during rcooval froo the mold 

HOT T EARS 

DESCRIPTION: 

A defect that affects both the SURFACE and the STRUCTURE of a casting. A 
hot tear is a cracklike defect that occurrs during solidification. As the 
molten raetal touches the comparatively cold mold surface, it soldifies as a 
skin and starts contracting ahead of the renainder of the casting. This 
skin is placed in tension end may tear if it becooos overstressed. These 
tears are characterized by having a heavily oxidized surface while CRACKS 
(see above), which occur after solidification, are relatively clean. 
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CAUSES 
DESIGN: 

* uneven or too rapid cooling 

- excessive mold rigidity restraining nornal contraction of the 
netal 

- poor pattern design 

too small a radius of curvature at a section boundary 
too high a ratio of areas between sections joined in a T 

INCLUSIONS 

DESCRIPTION: 

Inclusions are defects that arise due to foreign material trapped in a 

casting. inclusions arfect both the SURFACE and the STRUCTURE of a 

costing. Broadly speaking, inclusions are classified according to their 
origin. 

INCLUSIONS OF METALLIC ORIGIN (DROSS/SLAG) 

Improper melting and pouring practice may cause metallic inclusions in the 
casting. The formation of oxides, slag and other metallic waste oaterial 

is an inherent part of the melting process. This slag material is lighter 

than the molten metal and floats to the surface* Careful design of the 

piotiil feeding system attempts to take advantage of this fact to prevent 

slag material from entering the mold cavity, 

CAUSES 
DESIGN: 

- faulty gating design 
PRODUCTION: 

- poor quality control on raw materials 

- superheating of octal in melting furnace 

- melting cycle too long 

- turbulent flow of metal 

INCLUSIONS OF NONHETALLIC ORIGIN 

Somr of the possible sources of nonmetallic inclusions are: 

told and core material 

Extraneous Bold and core uterial may reoaln loose in the 
mold cavity or may be generated by the erosive action of 
the incoming molten metal. 

Pattern material 
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In investaont casting, residue from wax or plastic patterns 
may remain in the oold cavity and contaminate the casting- 
Crucible and furnace lining natcrial 

Lining breakdown in crucibles and melting furnaces adds 
contaminants to the oolten netal. 

(PARTING LINE) MISMATCH 

DESCRIPTION: 

A defect that alters both the SURFACE and the SHAPE of a casting. (Parting 
line) mismatch occurs in costing techniques that employ a two-piece mold. 
Misalignoont between the oold halves induces a step shift at the parting 
line of the casting. This results in a surface irregularity and loss of 
dimensional accuracy. The tolerable degree of parting line mismatch Is 
generally specified in the nechanical drawing of a casting. 

CAUSES 
DESIGN: 

- misalignment of pins and receptacles of the mold halves 
PRODUCTION: 

- play (due to wear over tine) developing between pins and 
receptacles of oold halves 

BISftUM 

DESCRIPTION: 

A SHAPE defect due to the incomplete filling of the mold cavity. Hisruns 

occur when an advancing stream of molten metal lacks sufficient force to 

overcome back pressure generated in the »old. (Typically, misruns occur In 

isolated thin sections of a casting.) 

CAUSES 
DESIGN: 

- faulty gating design 

(need for additional vonts, overflows or altered direction 
or iwtal flow) 

- low mold pemcability 

- incorrect mold/metai temperatures 
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PRODUCTION: 

- dirty metal (gating blockages) 

- failure of mold reaction inhibitors (excessive evolution of 

D&ses) 

- overlubrication (inhibits permeability) 

- pouring/injection rate too slow 

- inadequate quantity of pollen metal in shot well 

- mold temperature too low 

- metal temperature too low 

POROSITY 
DESCRIPTION: 

There is some confusion about the use of the term porosity. Vhen not 
distinctly referring to shrinkage porosity, porosity generally implies 
bubbles of gas entrapped in the metal during solidification. Many of these 
bubbles remain internal to the casting and thus represent STRUCTURE defects 
( internal porosity) . Certain others reach the surface during 

solidification and thus represent SURFACE defects (surface porosity) - 
Large surface porosity defects are commonly referred to as BLOWHOLES. 
Small surface porosity defects are commonly referred to as PINHOLES. There 
are three main sources of entrapped gas in a casting: 

1. Gas evolved within the molten metal itself. (Gas 
solubility in a metal decreases as temperature 
decreases.) 

2. Air trapped in mold cavity 

3< Gas evolved due to chemical reactions between the 
casting metal and tho mold or core material 
CAUSES 
DESIGN; 

- faulty gating design 

- ioproper mold reaction inhibitors 

- improper innoculants 

- low mold permeability 

* incorrect mold/metal temperatures 

- poor metal handling 
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PRODUCTION: 

* overlubricatlon 

- pouring/injection rale too slow 

- turbulent flow of »etal 

pouring/injection rate too high 
partial blockage in gating systoo 

- mold tenperaturc too high 

- mutdl tenperature loo high 

- contamination of raw materials 

SHRINKS 
(SHRINK CAVITIES. SHRINKAGE VOIDS, SHRINKAGE POROSITY, PIPE) 
DESCRIPTION: 

In cooling from a molten state to rooo temperature, metal goes through 
three stages of shrinkage: 

* volumetric shrinkage as a liquid 

- solidification shrinkage during conversion to a solid 

* contraction shrinkage as a solid cooling to room 
temperature 

Shrinkage defects (commonly referred to a shrink cavities, shrinkage voids, 

shrinkage porosity or pipe) are STRUCTURE defects. They arise due to metal 

shrinkage within a shell of already solidified metal that is not 

compensated for by a continued inflow of no 1 ten oetal. Shrinkage defects 

typically occur near the center of large, heavy sections of a casting. 

CAUSES 
DESIGN: 

- faulty gating design 

- poor control of direction of solidification 
PRODUCTION: 

- pouring/injection rate too slow 

- mold temperature to high 

- metal temperature too high 

- carbon equivalent of metal too low 
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MISCELLANEOUS HETAI/HOLD 1MTERACTI0M DEFECTS 
In tho discussion of green sand mold casting, we have already seen a table 

of miscellaneous surface defects that result from a poor metal -to-mold 
interface. In permanent mold casting techniques, there are similar 

defects. 

SOLDERING: 

Soldering is a SURFACE defect in permanent mold castings due to the 

adhering of metal to the mold surface. Soldering results in pimples or 

torn skin on the surface of the casting. It arises when the mold surface 

has becoae pitted or when there is inadequate lubrication between mold and 

metal. 

STAINS: 

Excessive lubrication of the mold surface can result in stains on the 

surface of the casting. 



